Ron Berman
@marketsensei
Followers
2,917
Following
849
Media
196
Statuses
3,488
Game theorist and data scientist. I study online advertising, marketing analytics and how curation algorithms impact platforms @Penn @Wharton .
Philadelphia, PA
Joined November 2010
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
THE LOYAL PIN THAI GL NO1
• 742403 Tweets
#ปิ่นภักดิ์EP1
• 732902 Tweets
Djokovic
• 351690 Tweets
Carlos Yulo
• 296527 Tweets
Rotherham
• 130672 Tweets
ヒロアカ
• 124913 Tweets
Austrália
• 98248 Tweets
エレちゃん
• 98143 Tweets
#instagramıGeriVer
• 79625 Tweets
ジャンプ
• 66759 Tweets
Caloy
• 64448 Tweets
Francisca Crovetto
• 58187 Tweets
TongrakMahasamut EP9
• 49678 Tweets
#10YearsWithEXOL
• 48693 Tweets
#EXOL10thAnniversary
• 47197 Tweets
最高のヒーロー
• 43386 Tweets
最高の物語
• 41122 Tweets
堀越先生
• 40888 Tweets
Nole
• 36242 Tweets
Byron
• 30633 Tweets
L1485 FOR LIFE
• 29049 Tweets
フワちゃん
• 27198 Tweets
BEST WISHES FOR YIBO BIRTHDAY
• 26171 Tweets
松山英樹
• 26115 Tweets
松山英樹
• 26115 Tweets
Fabiola
• 24425 Tweets
Scottie
• 24029 Tweets
Carlitos
• 19265 Tweets
尾田先生
• 13759 Tweets
ジョコビッチ
• 13323 Tweets
Federer
• 12540 Tweets
#そこ曲がったら櫻坂
• 11487 Tweets
Pinned Tweet
Companies invest a lot in analytics - but are these investments valuable?
@IsraeliAyelet
and I studied ~1,500 online retailers and found that using a descriptive dashboard increased their weekly revenues by 4%-10%.
>>
#MarTech
#BigData
#Analytics
#ecommerce
#DataScience
11
111
578
How are effects of online A/B tests distributed? How often are they not significant? Does achieving significance guarantee meaningful business impact?
We answer these questions in our new paper, “False Discovery in A/B Testing”, recently out in Management Science >>
22
200
1K
The paper is co-authored with Christophe Van den Bulte and analyzes over 2,700 online A/B tests that were run on the
@Optimizely
platform by more than 1,300 experimenters.
Link to paper:
Non paywalled:
>>
1
10
113
Generally, firms should try to test more radical variations in order to have larger impact on consumer behavior (we call this “swing for the fences”). Most variations will probably not be very impactful, but once in a while an innovation will prove to be very lucrative.
(Fin)
5
13
96
@Jeffely
Indeed the number of adults fully vaccinated in Israel has been quoted at 85% by officials.
This is what the age distribution of those recently testing positive in last month. Majority is from last two weeks. (Green: women, blue: men, y axis is age group).
2
9
72
Ordered coffee with regular milk in SF. Everyone gave me the "he's such a rebel" look.
5
1
67
Our analysis shows that in our data, hypothesis tests conducted with alpha=0.05 yield an FDR of 18%-25%.
Much higher than 5%.
That is, about 20% of significant effects chosen for implementation will not generate the business impact that was observed in the experiment.
>>
1
8
57
@mike_luca
@garjoh_canuck
@ronnyk
Yes. Just pick the outcome with the highest mean (Test & Roll) and you can use Latent Stratification if you really care about significance. Sometimes it helps.
On T&R:
On LS:
3
3
56
The paper is forthcoming in Marketing Science and is available at .
(Ungated version at )
>>
@MarketngScience
2
10
56
We then classify effects into “null” (zero) and “non-null” (pos or neg), to understand how many experiments, on average, have an underlying zero effect.
The answer is about 70%.
That is, 70% of effects will not show any impact on Engagement compared to a baseline.
>>
1
4
55
Traveling with a group of Penn Faculty on a Mission to Israel to build bridges with the Israeli academic community and bear witness to the impact of the Hamas attacks.
First visit since Oct 7.
Deep thoughts going through my mind rn.
#penn
#upenn
#pennfaculty
#mission
#israel
3
5
52
Been teaching digital marketing
@Wharton
for over 7 years.
Have had multiple students reach hundreds of thousands of social media followers in projects.
Still can't describe the (happy childish) feeling of this junior professor who reached 2,000 followers today.
Hi All! >>
1
1
49
First, we analyze the effects of all the A/B tests in our data. They are quite small. The median (and average) webpage variations have roughly zero effect on webpage Engagement.
But the distribution is quite long-tailed with some variations showing big effects.
>>
2
3
47
Congrats Wharton MBAs and Undergrads!!!
It was a pleasure to have taught you - keep in touch!
#WhartonGrad
#Wharton
@kwerb
@ShiriM528
@HummySong
@lynnwu02
0
3
47
If you're a quant marketer and am having trouble (like me) tracking all the conference deadlines etc., I created this list for us.
4
9
46
🚨Pre-doc Announcement🚨
@HummySong
and I are hiring a research associate to start in Summer 2022.
Interested in marketing analytics, causal inference, health economics and healthcare operations mgmt?
Come work with us
@Wharton
!
Details:
Please RT!
3
17
41
First, I coincidentally grab the table that a Nobel prize winner was eyeing at a coffeeshop.
Then, another Nobel prize winner shows up to my seminar talk.
Talk about a stressful day.
#stanford
3
0
36
Cue in TWFE DiD, Synthetic controls and what not.
Special quirk - our retailers have different time trends and potentially endogenous adoption timing.
Solution was a combination of
@jmwooldridge
’s POLS regression,
@ArkhangelskyD
et al. SynthDiD, and an IV.
2
3
33
First on the agenda - a meeting with the president of Israel, Mr. Isaac Herzog.
"You will meet a nation agonizing. An atomic bomb of emotions" the president told us.
#penn
#upenn
#pennfaculty
#mission
#israel
2
3
31
A big draw of the paper is that
@Optimizely
have graciously allowed us to publish the data we used in the analysis. We hope this would be valuable to other researchers as well.
>>
1
0
33
H2O blind tasting competition at Wharton marketing camp.
Who knew that Dasani tasted so good!
@alexoimas
@ShiriM528
@wdlrosa
@ZhenlingJ
@iyengar_raghu
4
0
33
Recently updated my CfP conference list. Some good conferences and opportunities to present coming up!
Let me know if I missed anything?
3
4
31
@shimrizzz
told us: "As a student your obligation is to always ask questions and not take anything for granted. Don't believe even me, ask questions and you'll see what evil has happened here and should never happen again."
Read more about this quote below
>>
2
8
31
alpha is the false positive rate (FPR), or
Pr(effect is significant | effect is null).
We care about the opposite,
Pr(effect is null | effect is significant),
which is called the false discovery rate (FDR).
>>
1
2
28
We use multiple methods to estimate the rate of true nulls and the FDR, but one was particularly fun to learn about, as it was developed to estimate false discoveries in genomewide studies. You can read about it here: .
>>
1
1
26
I know we always complain about reviewers, but dear authors, if you submit a paper which is over 45 pages and less than 1.5 spacing (on initial submission!) you are not helping the paper...
3
2
26
@HughJassole96
@JewishWonk
"did not happen in a vacuum" is like saying "but I can understand it" as in " I denounce the Holocaust, but it did not happen in a vacuum".
It's ok to say some things are just not ok. Ever.
6
0
22
As of today, mask requirements have been fully rescinded in Israel. Country fully open. So refreshing to be here.
0
0
23
I learned to make empanadas from my mom when I was a young, but we always bought the dough.
So I thought it's really hard to make, but my sister insisted it's easy and sent recipes.
So today's weekend experiment was a success!
#argentino
#vamoscarajo
4
0
23
Our paper estimates the business costs of these false discoveries, and discusses and tests possible solutions that firms can implement. The details are a bit beyond the scope of this thread, but we hope that the paper with the accompanying data and code will prove useful.
>>
1
1
22
Excellent thoughts on Behavioral Economics, from an actual Behavioral Economist...
0
0
22
Luckily, all these methods showed the same converging results, which are that adopting the descriptive dashboard yields an increase in revenues, diversity of products sold, number of repeat customers and number of transactions.
As usual, causality disclaimers apply.
>>
1
0
21
By comparing the results of dashboard users to non-users, we see that only users reap the benefit from the dashboard.
This allows us to rule out improved performance due to an unobserved and unrelated mechanism.
>>
2
0
20
I'm pretty sure they're using DMV employees as vote counters this election.
1
0
20
Often I find it hard to track conference deadlines and plan academic year travel.
Here's a live list of CfPs relevant to quant marketers.
Did I miss anything? Let me know...
#MarketingAcad
h/t:
@eleafeit
@dade_us
@AnikoOery
2
6
19
ngl, competition was tough, and the entries were amazingly tasty!
Turns out we have some amazing bakers in the marketing department.
Which one would you pick?
#whartonmarketing
#chillingcommittee
#bakeoff
@ebradlow
@Prof_Yildirim
@iyengar_raghu
@barbarakahn
@wdlrosa
2
0
19
As if figures and tables at end of working papers weren't bad enough, let me tell you about footnotes lodged at the end.
0
0
19
All are welcome!
Join us this Thursday May 6 at 12pm EDT for the
@mit_ide
lunch seminar series with guest
@marketsensei
of
@Wharton
discussing "False Discovery in A/B Testing"
Register free:
@sinanaral
@daveholtz
0
3
1
0
3
19
Major 2021 updates today to the list of call for papers for conference relevant to quant marketers.
0
2
19
Why are descriptive analytics so popular then?
Although they often leave users to generate their own insights, they provide a simple way to assess different decisions, enabling managers to extend the range of actions they can take and to integrate new technologies.
>>
1
1
19
Should we tell him?
How do you become a professor at a business school? I would like to wear a Tom Ford suit to work and teach 25 yr olds how to add fractions.
55
102
3K
0
0
18
You can learn a lot about consumer preferences in these times.
Clearly Trader Joe's customers didn't yet realize that Bamba is superior to anything else, especially in defeating plagues.
Also, why not smoked fish for a quarantine? Delish!
4
0
16
@ben_golub
Yes, it is used to measure attribution of online ads (and affects budget allocation and compensation of publishers):
1
0
18
@TaylorLorenz
I teach a pretty serious (and popular) course on digital marketing
@Wharton
and your reporting is tremendously helpful in analyzing case studies and learning how phenomena spread.
Please ignore the haters and keep up the great work...
0
0
16
Parents in law received first vaccine shot. Parents scheduled for next week.
This is happening!
0
0
18
Some of us are smiling, but we are in a mobile bomb shelter under rocket attack.
#penn
#upenn
#pennfaculty
#mission
#israel
2
2
17
Yay and congrats to
@eleafeit
, the best co-author that was and will ever be!
🎉🎉🎉
Time to celebrate! I just learned that the great
@eleafeit
recently got tenure at
@DrexelUniv
@LeBow
. She's a world-class scholar and I'm so happy for her!
9
1
76
2
0
17
@haneenshib
יש לי חבר שגר בארהב שכל פעם שהוא מתקשר לשירות לקוחות של משהו (נניח חברת תעופה או בנק) אם תוך כמה דקות לא נפתר מה שהוא רצה הוא פשוט מנתק (בלי לומר שלום) ומנסה שוב.
הוא גילה שיותר קל ליפול אקראית על מישהו שרוצה לעזור מלהסביר עצמך לאמריקאי שלא מבין כלום...
4
0
16
ארבע לפנות בוקר. יושב בשדה התעופה בסינגפור בדרך חזרה לפילדלפיה.
בטלויזיה CNN משדרת תמונות מתל-אביב של מדינה שהממשלה שלה מפוררת אותה.
ולי עצוב, ותחושת מחנק וחוסר אונים.
2
1
15
We used data from over 1,500 small and medium ecommerce global sellers (with mostly Shopify stores) with average monthly revenues of ~$60K.
Every retailer adopted an analytics dashboard that displayed KPIs such as weekly sales, avg basket size, conversion rate etc.
>>
1
1
16
Visiting communities in southern Israel to bear witness to the aftermath of the Oct 7 Hamas attacks.
This might be our most difficult and emotional visit on this trip.
Last time I was here was about 20 years ago. It also required bullet proof vests...
2
2
16
Take my money
My dream academic job: freelance scientific “closer.” You hire me to do skilled, unemotional revision work when you’ve completely lost the will to continue with a project, and I never have to start one.
176
336
8K
1
0
15
P.S. I didn't go into all details, minutiae, caveats and disclaimers in the paper. There is much more, and I would love for people to read it.
1
0
14
While preparing for a zombie apocalypse, it's important not to forget the little things.
#notmebaking
3
0
14
The concept of a true null is somewhat subtle (as we explain in the paper), since we often assume that true-nulls don’t exist.
However, as long as we test for a null hypothesis of a true null using significance testing, there is no reason to assume the null cannot be true.
>>
1
0
15
Are social media algorithms to blame for filter bubbles, online polarization and the shallowness of online content?
The answer is maybe, but probably not. All based on a recent paper with
@zsoltkatona
: “Curation Algorithms and Filter Bubbles in Social Networks”.
[Thread]
2
4
14
On New Year's Day, 2024, as war waged in Israel and Gaza, a group of professors from the University of Pennsylvania boarded an El Al flight to Tel Aviv. Representing fields as diverse as statistics, film, orthopedics, and law, they brought messages of friendship and support.
>>
Traveling with a group of Penn Faculty on a Mission to Israel to build bridges with the Israeli academic community and bear witness to the impact of the Hamas attacks.
First visit since Oct 7.
Deep thoughts going through my mind rn.
#penn
#upenn
#pennfaculty
#mission
#israel
3
5
52
1
2
14
Steve says it much better than I can.
Own it.
Why are so many university leaders hesitating to condemn these atrocities? Because they are scared of the very extreme "social justice" warriors who lack any sense of critical thinking. Those who claim that responsibility lies on "both sides" and who support Hamas actions \1
5
24
165
1
0
14
This is great news for anyone using social media for research.
Wish more platforms were also as doing this.
Academics are one of the biggest groups using the
#TwitterAPI
to research what’s happening. Their work helps make the world (& Twitter) a better place, and now more than ever, we must enable more of it.
Introducing 🥁 the Academic Research product track!
117
992
3K
1
2
14
Every single day
When you speak two languages and start losing vocabulary in both of them = byelingual
1K
172K
712K
0
0
14
As an Israeli this has been a unique honor and experience
First on the agenda - a meeting with the president of Israel, Mr. Isaac Herzog.
"You will meet a nation agonizing. An atomic bomb of emotions" the president told us.
#penn
#upenn
#pennfaculty
#mission
#israel
2
3
31
0
1
14
We would like to especially thank
@avicgoldfarb
who led a very constructive review process for the paper.
1
0
14
The stories of acquaintances whose family members were kidnapped by terrorists are heartbreaking.
A date which will live in infamy.
0
1
13
New on Google Scholar: "Save paper for reading later".
Finally.
So much interesting stuff to read though...
0
1
12
Newly out in
@PNASNews
. A 23 condition megastudy with 680k people to encourage flu vaccination.
Thanks for bringing us onboard
@katy_milkman
!
1
2
12
Instead, descriptive analytics serve to help retailers monitor additional marketing technologies (martech) and amplify their value.
Most retailers adopt additional technologies, but only the retailers that use the dashboard are able to benefit from them.
>>
1
1
13
@deaneckles
@causalinf
@eleafeit
and I have, in R, with both MLE and Stan models. If you can assume monotonicity (no defiers), you get upper and lower bounds with simple averages. For more identification, more assumptions are needed. Good reference, IMO:
1
4
13
Thanks for sharing my work with
@eleafeit
,
@StatModeling
.
It helps spreading the word...
“Test & Roll: Profit-Maximizing A/B Tests” by Feit and Berman
1
12
31
1
1
13
As PhD application season is in full gear, we are holding two information sessions about our PhD program.
For details:
0
5
13
Listen to
@dggoldst
. I trust him. :)
How to run a test & roll to maximize profit in A/B tests by Elea Feit and Ron Berman (
@eleafeit
&
@marketsensei
)
Handy calculator:
1
5
33
1
0
13
@ben_golub
@Andrew___Baker
@JlibDoesEcon
@danielluo_pi
Should be shared more often than that selection graphic.
1
0
13
A big logistical challenge with the Pfizer vaccine is apparently the shipment size (975 per tray). Once defrosted, they go bad quickly.
In Israel that has been the largest hurdle with distribution to small non-dense locations.
2
3
13
When the retailer adopted the dashboard, the dashboard’s provider also collected historical data, so we see performance before and after the adoption.
Because the dashboard was adopted in different times, the adoption is staggered, which you know what it means… 🙀🙀🙀
>>
1
0
13
JCR has only 30% preregistration? Seriously?
Colada[115] shows the prevalence of pre-registrations in papers published in 2022, with lessons on how to improve in 2023+
1
18
71
4
2
13
A very interesting analysis on the troubling news from Israel that the vaccine might seem ineffective against Delta.
Once the variant started spreading more evenly (unfortunately) we're starting to see how great the vaccine is.
Vaccine impact in Israel: changing trends. Analysis thread with
@AArgoetti
.
From about a month and a half ago we are experiencing a new “Delta” burst. Cases are accumulating, and more and more severe patients are in the hospitals.
20
245
678
0
3
12
A statistically significant result generated by a true null effect is called a false discovery. Although we often think of the significance threshold we set for hypothesis testing (e.g., alpha=0.05) as the rate of false discoveries, this is not actually what we get.
>>
2
0
13
Rightmost column - age group. Leftmost column - percent of age group vaccinated in Israel (first shot).
At current rate, the 60+ age group should be covered within 2-3 weeks.
1
1
13