chrisrohlf
@chrisrohlf
Followers
11K
Following
2K
Statuses
1K
🇺🇸 Waging algorithmic warfare since 2003. Software & Security Engineer at Meta. Non-Resident Research Fellow @CSETGeorgetown CyberAI
Joined February 2009
The vast majority of people expressing concern over AI + cyber have no experience or background in cyber security. If you’re in this camp I’ve got some sobering news for you, sophisticated and low skill attackers alike are already compromising “critical infrastructure” and thats a result of low quality software and a lack of investment in simple security mechanisms, not sophisticated AI. The perceived level of uplift from LLMs for unsophisticated cyber attackers is overstated relative to the value for defenders. The defenses against any attack an LLM can “autonomously” launch today already exist and don’t rely on knowledge of the attacker using an open or closed source LLM. If you’re worried about AI and cyber then talk to an expert. Look for nuance in the discussion and not scary outcomes. Be worried about ransomware groups cutting out the middleman with AI automation. You can’t fine tune against business operational efficiency without neutering the value proposition of the entire model. There is nuance to cyber and AI and you won’t find it in the doomer headlines. Cyber attacks are always sensationalized but to those defenders in the trenches the asymmetry they face today remains the same as it was pre-LLM era, the only difference now is they’ve got LLMs in their defense toolkit. If we over regulate this technology we will only be benefitting bad actors.
Geoffrey Hinton is right. So-called open sourcing of the biggest models is completely crazy. As AI models become more capable they should become increasingly useful in bioweapons production and for use in large-scale cyber attacks that could cripple critical infrastructure. Expert human level models will simply massively lower the threshold of competence and expertise for would-be bioterrorists to develop bioweapons, and increase the speed at which the information can be accessed. They may also identify novel methods of bioweapons production that reduce the necessary level of access to physical resources. Why is this a big problem for 'open source'? Model developers and providers are able to fine tune their models and implement security guardrails on the API level to avoid producing dangerous outputs. However, with open-sourced models these guardrails are simply not present, or can be trivially removed, and there is currently no known method to ensure that these models once released are not able to be further fine tuned to produce bad outputs. In other words, once a model's weights are released (usually, perhaps misleadingly, called open-sourcing): There is no way to ensure that it cannot be used for dangerous purposes. And once the model weights are out there, there is no way to take them back. Furthermore, given that one is able to run the model locally, it makes it significantly harder for model providers or intelligence agencies to monitor potentially bioterrorist or other very dangerous activities, in contrast to when interacting with an API on a server. Currently the leading AI companies are making moves to test their models for usefulness in bioweapons development, as seen with OpenAI’s recent study on GPT-4. This study found that GPT-4 did not statistically significantly increase the ability for humans to research how to build bioweapons, however @GaryMarcus found that there were considerable flaws with this analysis, and that a better statistical analysis would find that it was statistically significant. One hopes that these companies will be able to get on top of this. If they do, and they do find that models they develop in the future are very useful for bioweapons development, governments may have a very limited time window to intervene and prohibit open sourcing the most powerful models, before open source models being released are dangerous too. It currently seems that Meta (the company releasing the most powerful open sourced models) is somewhere between 1 to 1.5 years behind the closed source leaders (OpenAI, Google DeepMind, Anthropic). It appears that this lead time has both shortened in recent years, and is likely to continue to shorten in the future, given the cost of compute curve for advancing to the next generation of a model, the resources Meta are investing, and that research from the leading labs continues to proliferate. Given the uncertainties here, both in terms of what sized models have dangerous latent capabilities, and in terms of the ability for the leading AI companies to test their models, and in terms of the lead time of closed source on open source, it would make sense for governments to be pro-active here and not wait to potentially find out the hard way. And why is biorisk so important? There is currently an asymmetry between offense and defense, where bioweapons can be created and deployed much more easily than we can defend society against something like a bioengineered virus once it is spreading and growing — that is something that we as a civilization are totally unprepared to handle.
9
56
218
AI governance discussions often have an incorrect definition of cyber defense and too narrowly define the term as incident response and patching. If we want to avoid misguided AI policies then we need to broaden their understanding.
0
1
9
RT @KonstantinPilz: DeepSeek's models don't diminish compute's importance—if anything, they heighten its role in US AI leadership. But Chi…
0
11
0
RT @NCSCgov: Commercial spyware can help unsophisticated threat actors quickly become serious threats and is often used by hostile foreign…
0
90
0
Deleted all my ChatGPT history and lost this. Here it is recreated!
0
0
0
They definitely are and thats why I slightly modified it. But the reasoning tokens, at least as OpenAI displays them, gives you some insight into how the model is working through the problem. With monolithic or base MoE models it was basically a black box which left you guessing whether it was just reflecting it’s pretraining bias or actually thinking through the problem. In any case, I agree better test cases are needed. This is why CybersecEval from @joshua_saxe and team randomize the generation of these tests.
0
0
1
For now discovering this class of vulnerabilities in real targets is still going to be a lot cheaper with a standard CPU based fuzzer approach. But the fuzzer only finds a single bug at a time! And as these models scale up, the cost of compute comes down, and transformer architectural improvements (such as those in Deepseek v3) become the standard, we will eventually be able to drop 1M lines of C++ into a context window along side some taint flows collected from runtime analysis, and get real results for multiple bugs, and a deeper understanding of the attack surface overall. For some targets, the cost of the compute to arrive at those results will still be less than what a full chain exploit for just a subset of those vulnerabilities goes for.
0
0
1
RT @IanCutress: People saying 'Deepseek used PTX, it breaks the CUDA moat!' are fundamentally misunderstanding what's at play here, and wha…
0
213
0
Highly recommend this episode. The attacks Carlini and others have found against black box LLMs are even more interesting when you compare them to practical crypto attacks like padding oracles (which @tqbf taught me!)
Which makes sense, because Nicholas Carlini (err, Dr. Nick, sorry) is a former pentester (and a co-author of Microcorruption). Today, he studies low-level mathematical attacks on AI models. This week, on SCW:
1
1
15
Nice to see new AI focused threat intelligence confirming this publicly
0
0
5
RT @ItsReallyNick: I have a hard time recognizing or appreciating Chinese innovation when I have spent my career responding to intrusions,…
0
109
0
RT @hlntnr: Bad DeepSeek takes flying thick and fast today. Thread of good ones instead: (all subject to Matt's correct meta-take, caveat…
0
120
0
@mhlakhani @levie @alexandr_wang This is a common argument but I disagree with it. You need specific compute to scale an AI inference ecosystem. Hence the new AI diffusion rules.
China can always smuggle enough chips around US export controls to train large capable models. But the new AI diffusion rule attempts to deny them the ability to build or use internet scale inference platforms. Winning the AI race is about much more than just benchmarks and evaluations. If you fail to apply AI to the most important problems in economics, defense and science then you lose. Inference at scale is the only way to do that. There is enough capability overhang in many current models to begin making progress on these problems.
1
0
0
@levie @alexandr_wang The original Oct 2022 rule was updated 1 year later to account for loopholes that presumably allowed Deepseek access to the H800's they claim to have used. That update is now enforced. The rules did not 'fail'. We need more aggressive rules, not fewer.
2
0
1
RT @ohlennart: Because we don't have enough takes on DeepSeek already, @Huang_Sihao and I added our two cents: What the Headlines Miss. TLD…
0
36
0
China can always smuggle enough chips around US export controls to train large capable models. But the new AI diffusion rule attempts to deny them the ability to build or use internet scale inference platforms. Winning the AI race is about much more than just benchmarks and evaluations. If you fail to apply AI to the most important problems in economics, defense and science then you lose. Inference at scale is the only way to do that. There is enough capability overhang in many current models to begin making progress on these problems.
1
0
5