alex peysakhovich 🤖
@alex_peys
Followers
5,337
Following
789
Media
139
Statuses
1,191
Explore trending content on Musk Viewer
JUNGKOOK
• 1321507 Tweets
Celtics
• 283011 Tweets
sabrina
• 147772 Tweets
Boston
• 143837 Tweets
Mavs
• 135616 Tweets
Dallas
• 128664 Tweets
Luka
• 124810 Tweets
ariana
• 100147 Tweets
Kyrie
• 93795 Tweets
Porzingis
• 83946 Tweets
Tatum
• 83608 Tweets
Dr. Phil
• 58182 Tweets
ナイトワンダラー
• 30026 Tweets
#スターレイル予告番組
• 28548 Tweets
#PowerGhost
• 26354 Tweets
#BiharSpecialStatus
• 23800 Tweets
Hayırlı Cumalar
• 23309 Tweets
Miss Alabama
• 21067 Tweets
Kaytranada
• 15969 Tweets
#يوم_Iلجمعه
• 15843 Tweets
Tariq
• 14144 Tweets
PRABOWOingin RAKYATmakmur
• 14092 Tweets
SalingJAGA SalingBANTU
• 13461 Tweets
GIGIL OUT NOW
• 12945 Tweets
ドラマCD
• 11715 Tweets
Carlos Miguel
• 11108 Tweets
ジョングク
• 10792 Tweets
really love when authors do things like explain equations in-line with colors etc... just makes papers so much easier to read.
40
338
4K
once did a screening interview with a “famous ml hedgefund”. got a leetcode-style problem to find connected components in a graph. they wanted bfs or dfs or whatever. i took the svd of the laplacian and counted the number of 0s. they didn’t pass me because “you can’t use numpy”
32
68
2K
standard ml: oh no my model is memorizing the training set, better add some regularization to make that not happen
llm ml: ugh it’s hallucinating, why can’t it just memorize some of the training set
10
33
336
facebook didn't keep it secret, here is the paper that explains the early comment ranking system for exactly this issue
Engineers have figured out to cut back on the toxicity of the internet — but firms who are good at it keep it secret as a competitive advantage.
A fascinating question of private vs. public interests.
9
10
152
4
37
296
@Apoorva__Lal
you can also estimate null space dimension by “uniformly” randomly sampling vectors, hitting them with the matrix, and seeing what % are 0. no numpy required
2
1
202
ran gpt4 128k context on the "1 useful document + K distractors" from our "attention sorting" paper, seems like the very long (more than 32k) doesn't work that well. 32k is still extremely impressive though!
also claude2 clearly has some nice trick behind the scenes
9
27
203
i worked at fb for 9 years, this is the first time i have ever seen people excited about an fb original product at launch time
7
7
192
the foundation of machine learning is that inside every big matrix lives a much smaller matrix and you should just use that
Text embeddings are the unsung heroes of making ai work. Can't believe we found a way to make language into numbers, the matrix stuff backprop etc afterwards feels almost straightforward once it's done. All tricks are downstream of this.
With numbers you can add them (merge),…
8
13
117
7
16
193
this is just proof that agi is achieved, we can now simulate a real software engineer perfectly
Interesting watch. In an official Devin demo, Devin spent six hours writing buggy code and fixing its buggy code when it could have just ran the two commands in the repo's README.
5
19
280
1
11
158
there is a huge gain to be made by every company on the planet by just running every file in their codebase through gpt4 with the prompt "what's wrong with this?"
GPT-4 "discovered" the same sorting algorithm as AlphaDev by removing "mov S P".
No RL needed. Can I publish this on nature?
here are the prompts I used
(excuse my idiotic typos, but gpt4 doesn't mind anyways)
96
449
3K
3
10
151
one of the reasons i left behavioral science (much happier now anyway, so it all worked out!) was that it was pretty clear many people were bs-ing and when getting jobs/tenure/etc... depends on # papers published, it is very hard to compete with people willing to make stuff up
7
8
133
lots of us are too busy working on this stuff to get in these debates on twitter, but yann is completely right here (just like he was with the cake thing, and the neural net thing, and and and)
The heretofore silent majority of AI scientists and engineers who
- do not believe in AI extinction scenarios or
- believe we have agency in making AI powerful, reliable, and safe and
- think the best way to do so is through open source AI platforms
NEED TO SPEAK UP !
170
408
2K
4
4
127
tfw one of your paper techniques gets rediscovered. now i know how everyone that wrote papers in machine learning between 1980 and 2012 feels
1
4
111
evergreen question: how does anyone ever do any data cleaning in python? pandas is the worst thing i've ever worked with compared to e.g. tidyverse
18
2
96
this is the paper btw: you should also read it, it's cool
0
4
96
@tszzl
i find these salaries surprisingly low given the value provided and the working conditions. if you scale cardiac surgeon to 40 hours per week that's ~400k, that's like an L6 at google/fb....
8
1
88
hype cycle for llms is moving so fast and everyone is at totally different points on the curve, it's wild
5
22
86
ok, it's after 5pm, i am no longer a facebook employee, i will miss FAIR and my awesome colleagues, but it's on to new things!
also, now everyone can stop asking me to help their weird uncle get unbanned, you know what they did.
14
0
87
the best conversations i've had about "ai risk" have been with integrity people from social media companies, they have the most experience with having complex systems behave in unintended ways and also deal with internal pressures eg. incentives of other teams to get metrics up
5
9
85
authorship norms differ a lot across fields...
cs: "oh, we talked about this at lunch for 5 min, you should be a coauthor!"
econ: "you spent weeks in a library collecting data and you want to be a coauthor? gtfo!"
...you can guess which one creates more collegial atmosphere
Daron Acemoğlu and David Card on undergraduate vs. graduate research assistants:
43
306
3K
3
3
79
me: oh what have you been up to today?
gf: not much, just chilling *casually drops 4 nature/science papers in one day*
New in Science and Nature: The first four papers from the U.S. 2020 Facebook and Instagram Election Study!
5
40
172
0
2
79
linear algebra is the basis for everything that works
3
6
80
the work on fine tuning llms in parameter efficient ways that is coming out is just so cool and clever. really shows the power of open source + flexible frameworks that allow you to easily write model blocks and just type loss.backwards()
0
19
80
i stared at this for probably 90 seconds thinking “what else could it be other than epsilon?” had to check replies for the answer
The look my wife game me when I immediately shouted “epsilon”, without really thinking it through…
225
565
25K
4
3
77
its crazy how data inefficient neural net optimization can be - i have a problem where a linear regression gets 80% accuracy but it takes 400k samples and 100+ epochs of training for a 2 layer relu net to match that (my learning rate is fine thanks for asking)
5
4
65
heuristic: if you’re talking about bias-variance tradeoffs you’re doing classic machine learning, if you’re saying “crap i need more gpus”, you’re doing post-modern machine learning
Recently I heard someone call it ��classic machine learning”
Like damn bro that hurts
36
16
350
1
4
61
explaining something i'm working on to a jr colleague, he says "oh you're one of those last generation ai people that actually knows math, that's interesting"
3
3
60
one simple statistical hack to improve your language model evaluations: don't take mean(model 1 performance) and compare it to mean(model 2 performance). instead, consider the *paired* statistic mean(model1_acc(question i) - model2_acc(question i)). in many tasks is huge latent…
2
4
59
@GrantStenger
agree that in theory bfs/dfs is better but if you compute rarely (are you decomposing 10m vertex graph every second? why?), it doesn’t really matter what you use as long as it runs. we did randomized svd on huge graphs all the time at fb and it was fine.
2
0
58
"a man painting a horse" by stable diffusion. it just couldn't decide how to parse that sentence so it just did both
5
7
54
one reason why “big fraud” is so prevalent in behavioral science is that there are few real world checks, other sciences have more downstream engineering built on them and so it is easier to filter fake effects (not perfect, plenty of bad examples in “harder” science too)
6
3
53
europe: we are the world champions of shutting down innovation with weird regulation
california: hold my beer
The California senate bill to crush OpenAI's competitors is fast tracked for a vote. This is the most brazen attempt to hurt startups and open source yet.
🧵
31
153
481
1
16
50
@aryehazan
random matrix theory and various concentration results are all extremely unintuitive (at least to me).
1
2
49
been playing with the
@huggingface
mteb leaderboard () all day, super interesting dataset with very interesting correlation pattern across tasks. if you're good at one retrieval, you're good at all of them. other stuff? much less predictable
2
9
45
gemini won’t draw me because im a stereotype apparently.
1
5
46
this is the correct take. llm are the glue code that will allow us to put so many other technologies together. if you think of the llm as originating in machine translation this shouldn't be too surprising - they're exactly great for translating between many different modalities
For people not paying close attention to AI right now, all the pieces are coming together at the same time to rapidly transform how we live and work.
Here Stanford robotics researchers demonstrate GPT-4 control of a robot which can begrudgingly follow your spoken instructions:
5
10
60
3
8
43
neural network architectures should just copy whatever corvid brains are doing
@norabelrose
There’s also at least some tasks on which performance scales linearly with log pallial neuron count.
5
4
49
1
2
41
if i'm going to complain about companies not releasing their models/model data, i should give credit
@MosaicML
has a nice release of mpt with full transparency. have been playing with the instruct model and it's pretty impressive!
2
2
43
current programming workflow:
1) ask chatgpt to write code to do X
2) remove comments from code and see if it runs
3) pass code back into chatgpt, ask to explain "to an idiot" what the code does
if result of step 3 matches X, i assume it's right and move on to the next part
3
2
41
i loved doing my phd. when i saw the academic marker afterwards i noped out but the phd itself was super fun, i had great advisors (one of whom won a nobel during when i was his student and still met with me that week to discuss an experiment i was running) and learned a ton
there's a lot of shit talking grad school on here but genuinely i loved grad school, it was amazing, and i am grateful to the
@CornellSoc
program for the opportunity
0
0
8
2
0
42
let's play the "when did the vision pro presentation start?" game
1
1
40
awesome paper. contains great bangers like: "When we wonder whether the machine is sentient, the machine’s answers draw on the abundant science fiction material found in its training set."
0
13
37
this is how i feel abut state space models vs transformers
0
3
38
this is literally the point of the technology
1
1
37
man it's crazy how there are no more photographers since digital cameras and photoshop came along
6
3
36
the last 3-6 months in ai have really changed my baseline on a lot of things from "i kind of understand some parts of the world" to "i don't fucking know what's going to happen, predicting the future is impossible"
2
0
35
all problems in life can be solved if you realize that within every really big matrix hides a much smaller matrix that preserves most of the information
Thanks to parameter-efficient finetuning techniques, you can finetune a 7B LLM on a single GPU in 1-2 h using techniques like low-rank adaptation (LoRA).
Just wrote a new article explaining how LoRA works & how to finetune a pretrained LLM like LLaMA:
29
297
2K
0
2
36
the fact that openai didn't release an april fools statement admitting that gpt3 was just 10,000 people in a call center is a bit disappointing
1
1
36
also if you do this with social science analyses files you can unpublish like 50% of all papers
1
2
36
given the way some of these detectors tend to work (look at whether something is highly likely under the model) it seems like any document that the model has memorized will trigger the detector as "AI generated"
someone used an AI detector on the US Constitution and the results are concerning.
Explain this, OpenAI!
460
3K
36K
2
2
34
another for the “you should be suspicious if your machine learning is too good” pile
Machine learning predicts hit songs from brain responses with 97% accuracy. Self-reported liking isn’t predictive. 😮
40
163
1K
2
1
34
everyone who did substantial work on a paper should be an author, if you're not sure whether someone's contribution should count as substantial, you should err on the side of including them. doing otherwise is wrong. don't @ me you won't convince me otherwise
2
0
28
llm development went from "release paper with full details" to "release model evaluations but not training details" to "here are some videos" real quick
Introducing 100K Context Windows! We’ve expanded Claude’s context window to 100,000 tokens of text, corresponding to around 75K words. Submit hundreds of pages of materials for Claude to digest and analyze. Conversations with Claude can go on for hours or days.
218
1K
5K
2
2
30
"computers can't do exploratory data analysis like people" is the new "computers can't play chess like people"
3
1
29
@cauchyfriend
i don’t know what a class is in python and i worked on some of the most used internal and user facing things at fb for 9 years
2
0
27
playing in the text embedding space of sd turbo is pretty fun. this is just taking convex combination of 2 prompts
1
1
25
this is not a randomized experiment. the much more likely story here is that twitter is better at *figuring out* which of two papers with similar abstracts/conference accepts will be important later, not that twitter *causes* it
Crazy AF. Paper studies
@_akhaliq
and
@arankomatsuzaki
paper tweets and finds those papers get 2-3x higher citation counts than control.
They are now influencers 😄 Whether you like it or not, the TikTokification of academia is here!
64
285
2K
2
0
27
where are all these ai people getting time to go to meetups, make flashy videos, etc…? between cleaning data, writing code, and watching the training runs i don’t even have time to cherry pick outputs to post “we’re so back” threads on twitter
2
1
26
excited to finally drop a paper about an idea
@adamlerer
and i have been messing around with for a while
tldr: in a simple qa task, re-sorting documents in llm context by attention it pays to them, and *then* generating improves accuracy a bunch 1/n
1
6
27
oh academia
reviewer: you studied X but really you should have studied Y, reject
reply: yes Y is important, but X is also a thing with a huge literature (theory) + many companies trying to solve it (practice), is there something wrong with how we studied X?
reviewer: no, reject
3
1
26
science naming conventions have changed a lot…
physics in 50s: “we use a feynman parametrization to solve a model of heisenberg’s uncertainty principle and evaluate it on data from the oil drop experiment”
modern ml: “we attach a yolo model to a llama backbone and then look at…
0
2
26
this is your daily reminder that pandas indexing was created in the lower levels of hell to torture people trying to do basic things like concatenate
3
0
26
@johnpdickerson
weaknesses
math is hard
questions
why math so hard?
ethics review flags
why make me read math? just waterboard me already
1
0
25
fell into classic trap of spending 1 hour+ to automate something i could have done in 20 boring minutes
2
0
26
this board is a scam, why are there random ceos of oil and airplane companies, only a few legit scientists, and nobody from meta?
This morning the Department of Homeland Security announced the establishment of the Artificial Intelligence Safety and Security Board. The 22 inaugural members include Sam Altman, Dario Amodei, Jensen Huang, Satya Nadella, Sundar Pichai and many others.
310
245
1K
1
0
25
tired: all embeddings are data compression
wired: all data compression is an embedding
this is wild — kNN using a gzip-based distance metric outperforms BERT and other neural methods for OOD sentence classification
intuition: 2 texts similar if cat-ing one to the other barely increases gzip size
no training, no tuning, no params — this is the entire algorithm:
152
1K
7K
1
2
25
neat guide. hits one of my favorite metaphors: "statistics is like baking, you need to follow the recipe exactly, ML is like cooking, you need to constantly taste and adjust spices"
Self-supervised learning underpins today’s cutting-edge work across natural language, computer vision & more — but it’s an intricate art with high barriers to entry.
Today we're releasing the SSL Cookbook, a practical guide for navigating SSL + contributing to this space ⬇️
29
168
711
2
7
24
many people in ai want machines that have “general” intelligence - i don’t care (or believe that will happen) - i want dumb machines that free people from doing the boring, repetitive, and/or dangerous tasks that take away time from actual interesting pursuits
1
3
24
so i pasted the first half of an analysis script for one of my papers into gpt and asked "what is the person writing this trying to do?" 🧵 1/n
1
2
23
ugh these econometricians, everyone knows that the more layers your neural network has the more causal it is.
This is a common misconception I see a lot in my intro econometrics class. To detect causality in regressions you actually need to look at the *adjusted* R-squared, since the regular R-squared always increases with more controls. Hope this helps!
48
23
450
2
0
23
if you want to understand ml/ai, you can't do it by reading papers. being one with the matrix requires the terrible grind of building models to do stuff, getting frustrated when they don't work, working in the data mines, etc...
3
2
23
random projections for dimensionality reduction are bullshit and shouldn't work but they do
6
1
22
now i do ai and my day is mostly pasting pytorch errors into chatgpt
0
0
22
@Apoorva__Lal
im actually curious now: is there a fundamentally easy way to estimate size of null space from appropriately chosen observations (x, Ax)?
@ben_golub
nerd snipe here
0
0
21
cs researchers: why are people sending so many papers to conferences these days? we really should take actions to increase the signal / noise ratio
also cs researchers: this grad student went to the bathroom and didn’t come out with a neurips paper, i dunno…
0
1
22
i have a theory that zuck's metaverse obsession was always just a long, petty, con to get apple to do something dumb
0
1
21
0
0
20
i did 1 hike on the ca coastline and came up with at least 5 research things i want to try, time off is really good for the brain
3
0
21
a great way to see a major weakness of current llms is to take a task it can do an easy version of (sort these 2 words into alphabetical order) and watch it fail on a complicated version that could, in principle, be done by repeated application of the easy thing.
4
2
21
the impossible text to image prompt (images from DALLE, MJ v6, SDXL respectively)
2
1
21
best documentary ive ever seen
0
1
21
come join tech where nobody cares if you just type in all lower case without punctuation or grammar
In your application letter for
#PhD
/ postdoc, NEVER ever say:
"Hi prof"
"Hello"
"Dear Professor"
"Greetings of the day"
If you do, your email will be immediately deleted by 99% of professors.
▫️
Only start your applications with “Dear Prof. [second_name],”
And don’t…
289
337
3K
1
0
21
the main parallel to be drawn between oppenheimer-era physics and today's ai is that you succeed when you reduce your problem to matrix multiplication
2
2
19
ah yes stripe, the company famously founded, incubated, and grown in *checks notes* san francisco, france
This morning I met with
@collision
, President of
@stripe
– a European success story in payments.
Great conversation about the international payments landscape, how we tackle cyber-risks and fraud, and the role of financial education.
182
14
305
0
0
19
symbolic reasoning, 'i dunno just predict the next word of the internet', and bayesian methods
2
2
20
academics: oh no you copied a short description of something without citing the author, that's plagiarism
programmers: gotta write some code, ok let's find something someone else did and copy paste as much of that as we can
1
3
20
omg claude is now doing the "i don't wanna give you all the code" thing... please
@AnthropicAI
don't do this
1
1
20
this is just a chart of how long you wait for the server at the restaurant
Which country has the best food? A revealed preference approach.
112
186
765
1
2
19
this video is amazing, a VERY pretty explanation of “why the hell is there a pi in the Gaussian distribution?”
‘Metrics friends: I had never truly appreciated why pi shows up in the normal distribution’s pdf until just now.
Did you folks know this all along?! It’s so beautiful! Learned it from this 3Blue1Brown animation 🥹:
7
42
333
0
2
19
@ben_golub
its just one of many ways of assigning the total value (predictive accuracy) to members of a coalition (input features) when the value function is complex. it happens to be one that's relatively easy to compute. it's not that dumb, but its also only useful in a relatively small %…
1
0
19
in the long run, everything becomes ggplot
1
1
19
smart is overrated (and not just in econ)
Just ran across this hilarious essay by D. McCloskey about economist compliments. Feel like I should have heard of this before now:
4
11
64
0
1
18
python is simultaneously the biggest accelerator of and the biggest hurdle to progress in machine learning
llama-cpp-python requires pydantic 2.0.1, explicitly won't work with <=2.0
fastapi and chromadb requires pydantic 1.9.2, explicitly won't work with >=2.0
oh, dear...
@abetlen
30
14
166
2
2
18