Some text data is private & cannot be shared... Can we generate synthetic replicas with privacy guarantees?🤔
Instead of DP-SGD finetuning, use Aug-PE with inference APIs! Compatible with strong LLMs (GPT-3.5, Mistral), where DP-SGD is infeasible.
🔗 [1/n]
Excited to see the release of the book 🥳 and grateful for the opportunity to contribute a chapter. Big thanks to the three editors!
@pinyuchenTW
@LamMNguyen3
@nghiaht87
Happy to share the release of the book "Federated Learning: Theory and Practice" that I co-edited with
@LamMNguyen3
@nghiaht87
, covering fundamentals, emerging topics, and applications. Kudos to the amazing contributors to make this book happen!
@ElsevierNews
@sciencedirect
Super excited to set up the LLM safety & trustworthiness leaderboard on Huggingface, and we will keep adding new safety perspectives. Here we evaluate (open & close) LLMs and compressed LLMs. Looking forward to more exciting evaluations to assess and enhance LLM safety!!! 🥳
📣 Call for Papers 📣
We are now accepting submissions for our
#KLR
workshop at
#ICML2023
! Submit your completed research or work-in-progress at the intersection of knowledge reasoning🤔 and ML🤖 by May 26, 2023 (AOE)!
Stay tuned for the schedule🔥🚀
4 steps: 1) first draw random samples from an LLM, and then iteratively improve them by 2) computing the nearest neighbor, 3) selecting (with DP) the most similar ones to the private dataset, and 4) querying the LLM to generate more of such related samples. [2/n]
Aug-PE is compatible with open-source LLMs, where DP finetuning is hard to implement due to the need to calculate the per-sample gradient in DP-SGD. [4/n]
Aug-PE (w/ inference only) can match DP finetuning with the same model under privacy budget epsilon = 4, 2, 1 in some cases, and outperform it with stronger closed-source LLMs like GPT-3.5, where DP finetuning is infeasible! [3/n]
Aug-PE (w/ inference APIs) is more efficient than DP-SGD finetuning on the Yelp dataset. The running time of Aug-PE mainly depends on the number of API calls (associated with the hyperparameter L). [5/n]
@PandaAshwinee
Thanks! We exactly used downstream objectives to evaluate syn data. For Yelp/OpenReview syn data, we evaluate accuracy on downstream classification tasks. For PubMed syn data, we evaluate next-word prediction accuracy of a downstream model, inspired by