Ning Yu 于宁


Research Scientist
Netflix Eyeline Studios
ningyu.hust at gmail.com

Google Scholar| GitHub| LinkedIn| Twitter

News| Publications| Patents| Awards Mentoring| Chairing| Reviewing| Teaching

About me


I am a research scientist at Netflix Eyeline Studios, leading efforts in visual and multimodal generative AI. My research aspirations lie in controllable and customizable recreation of the world via generative AI, based on representations of multimodal contents and interactions between users and models.

I hold a joint Ph.D. from the University of Maryland and Max Planck Institute for Informatics where I was supervised by Larry Davis and Mario Fritz. I previously worked with Salesforce, NVIDIA, and Adobe.

My selected works include UniControl, X-InstructBLIP, ULIP-2, LayoutDETR, GlueGen, HIVE, Contrast and Attention GANs, Scalable GAN Fingerprints, Artificial GAN Fingerprints, GAN Fingerprints, Inclusive GAN, and Texture Mixer. My research was featured in media press such as Salesforce Newsletter, AlmostHuman, AIera, JiangMen, QbitAI, innovations report, and FoodPlus.

I am constantly dedicated to academic services, such as serving as area chair for CVPR, ICCV, ECCV, NeurIPS, ICLR, AAAI, AISTATS, WACV, and BMVC.

I am a recipient of Twitch (Amazon) Research Fellowship, Microsoft Young Fellowship, Qualcomm Innovation Fellowship Finalist x2, and SPIE Best Student Paper Finalist.

News


[2024/03] SimSCOOD is accepted to NAACL 2024. The code is released.
[2024/02] HIVE and ULIP-2 are accepted to CVPR 2024.
[2024/01] Model-Agnostic Hierarchical Attention for 3D Object Detection is accepted to ICRA 2024.
[2023/11] AnchMark is accepted to NeurIPS Workshop on Regulatable ML 2023.
[2023/09] UniControl is accepted to NeurIPS 2023. The code is released.
[2023/07] GlueGen is accepted to ICCV 2023. The code is released.
[2023/05] DE-FAKE is accepted to CCS 2023. The code is released.
[2023/04] Generated Graph Detection is accepted to ICML 2023. The code is released.
[2023/04] SPAD and Prototype LTR are accepted to IJCAI 2023.
[2023/04] RoSteALS is accepted to CVPR Workshop on Media Forensics 2023. The code is released.
[2023/02] Mask-free OVIS and Cont-Steal are accepted to CVPR 2023.
[2022/10] UnGANable is accepted to USENIX 2023. The code is released.
[2022/08] FcF Image Inpainting is accepted to WACV 2023. The code is released.
[2022/07] I am one of the top reviewers and serve as a session chair for ICML 2022.
[2022/07] RepMix is accepted to ECCV 2022. The code is released.
[2022/05] Vision-Language for Food Nutrient Estimation is accepted to Food Chemistry 2022.
[2022/04] Multi-Exit Privacy is accepted to CCS 2022. The code is released.
[2022/04] The code for RelaxLoss is released.
[2022/01] The code for Scalable GAN Fingerprints is released.
[2022/01] Scalable GAN Fingerprints and RelaxLoss are accepted to ICLR 2022 as Spotlight.
[2021/10] VIDNet is accepted to BMVC 2021. The code is released.
[2021/08] I defended my Ph.D. The dissertation about Human-Centric Deep Generative Models is online.
[2021/08] CV for Food Nutrient Estimation is accepted to Food Chemistry 2021.
[2021/07] Artificial GAN Fingerprints is accepted to ICCV 2021 as Oral. The code is released.
[2021/07] Dual Contrastive Loss and Attention for GANs is accepted to ICCV 2021.
[2021/05] CV for Food Nutrient Prediction is accepted to Food Research International 2021.
[2021/04] Re-Synthesis for Deepfake Detection is accepted to IJCAI 2021. The code is released.
[2021/03] AI for Food Nutrient Estimation is accepted to Journal of Food Composition and Analysis 2021.
[2021/02] Hijack-GAN is accepted to CVPR 2021. The code is released.
[2020/08] Class Balanced Experts for Long-Tailed Recognition is accepted to GCPR 2020. The code is released.
[2020/06] Inclusive GAN is accepted to ECCV 2020. The code is released.
[2020/03] GAN-Leaks is accepted to CCS 2020. The code is released.

Publications


Reference-Based 3D-Aware Image Editing with Triplane

Bahri Batuhan Bilecen, Yigit Yalin, Ning Yu, Aysegul Dundar

arXiv 2024

pdf project code



Text2Data: Low-Resource Data Generation with Textual Control

Shiyu Wang, Yihao Feng, Tian Lan, Ning Yu, Yu Bai, Ran Xu, Huan Wang, Caiming Xiong, Silvio Savarese

ICLR Workshop on PML4LRS 2024

pdf code



Shadowcast: Stealthy Data Poisoning Attacks Against Vision-language Models

Yuancheng Xu, Jiarui Yao, Manli Shu, Yanchao Sun, Zichu Wu, Ning Yu, Tom Goldstein, Furong Huang

ICLR Workshops on DPFM, PML4LRS, SeT LLMs 2024

pdf project code



RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content

Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, Bo Li

arXiv 2024

pdf



C-RAG: Certified Conformal Risks for Retrieval-Augmented Language Models

Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, Bo Li

arXiv 2024

pdf



HIVE: Harnessing Human Feedback for Instructional Visual Editing

Shu Zhang*, Xinyi Yang*, Yihao Feng*, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, Ran Xu

CVPR 2024

pdf project code



ULIP-2: Towards Scalable Multimodal Pre-training For 3D Understanding

Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese

CVPR 2024

pdf code blog press



SimSCOOD: Systematic Analysis of Out-of-Distribution Behavior of Source Code Models

Hossein Hajipour, Ning Yu, Cristian-Alexandru Staicu, Mario Fritz

NAACL 2024

pdf code



Model-Agnostic Hierarchical Attention for 3D Object Detection

Manli Shu, Le Xue, Ning Yu, Roberto Martín-Martín, Juan Carlos Niebles, Caiming Xiong, Ran Xu

ICRA 2024

pdf



Generated Distributions Are All You Need for Membership Inference Attacks Against Generative Models

Minxing Zhang, Ning Yu, Rui Wen, Michael Backes, Yang Zhang

WACV 2024

pdf code



X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning

Artemis Panagopoulou, Le Xue*, Ning Yu*, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, Juan Carlos Niebles

arXiv 2023

pdf project code press



Continual Adversarial Defense

Qian Wang, Yaoyao Liu, Hefei Ling, Yingwei Li, Qihao Liu, Ping Li, Jiazhong Chen, Alan Yuille, Ning Yu

arXiv 2023

pdf code



AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors

You-Ming Chang*, Chen Yeh*, Wei-Chen Chiu, Ning Yu

arXiv 2023

pdf code



On the Proactive Generation of Unsafe Images From Text-To-Image Models Using Benign Prompts

Yixin Wu, Ning Yu, Michael Backes, Yun Shen, Yang Zhang

arXiv 2023

pdf



LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer

Ning Yu, Chia-Chih Chen, Zeyuan Chen, Rui Meng, Gang Wu, Paul Josel, Juan Carlos Niebles, Caiming Xiong, Ran Xu

arXiv 2023

pdf project code BannerGen lib blog



UMDFood: Vision-language models boost food composition compilation

Peihua Ma*, Yixin Wu*, Ning Yu, Yang Zhang, Michael Backes, Qin Wang, Cheng-I Wei

arXiv 2023

pdf



UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild

Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, Ran Xu

NeurIPS 2023

pdf project code Huggingface space poster blog press



AnchMark: Anchor-contrastive Watermarking against Generative Image Modifications

Minzhou Pan*, Yi Zeng*, Xue Lin, Ning Yu, Cho-Jui Hsieh, Ruoxi Jia

NeurIPS Workshop on Regulatable ML 2023

pdf



GlueGen: Plug and Play Multi-modal Encoders for X-to-image Generation

Can Qin, Ning Yu, Chen Xing, Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, Ran Xu

ICCV 2023

pdf project code poster blog



DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Diffusion Models

Zeyang Sha, Zheng Li, Ning Yu, Yang Zhang

CCS 2023

pdf code



Generated Graph Detection

Yihan Ma, Zhikun Zhang, Ning Yu, Xinlei He, Michael Backes, Yun Shen, Yang Zhang

ICML 2023

pdf code



Detecting Adversarial Faces Using Only Real Face Self-Perturbations

Qian Wang, Yongqin Xian, Hefei Ling, Jinyuan Zhang, Xiaorui Lin, Ping Li, Jiazhong Chen, Ning Yu

IJCAI 2023

pdf code slides video



Learning Prototype Classifiers for Long-Tailed Recognition

Saurabh Sharma, Yongqin Xian, Ning Yu, Ambuj Singh

IJCAI 2023

pdf code slides video



Mask-free OVIS: Open-Vocabulary Instance Segmentation without Manual Mask Annotations

Vibashan VS, Ning Yu, Chen Xing, Can Qin, Mingfei Gao, Juan Carlos Niebles, Vishal Patel, Ran Xu

CVPR 2023

pdf project code poster blog



Can’t Steal? Cont-Steal! Contrastive Stealing Attacks Against Image Encoders

Zeyang Sha, Xinlei He, Ning Yu, Michael Backes, Yang Zhang

CVPR 2023

pdf code poster video slides



RoSteALS: Robust Steganography using Autoencoder Latent Space

Tu Bui, Shruti Agarwal, Ning Yu, John Collomosse

CVPR Workshop on Media Forensics 2023

pdf code Huggingface space poster



UnGANable: Defending Against GAN-based Face Manipulation

Zheng Li, Ning Yu, Ahmed Salem, Michael Backes, Mario Fritz, Yang Zhang

USENIX 2023

pdf code poster press



Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand

Jitesh Jain*, Yuqian Zhou*, Ning Yu, Humphrey Shi

WACV 2023

pdf project code demo Huggingface space poster slides
tech transfer: Photoshop 24.5 Remove Tool



Unsupervised Dense Retrieval Deserves Better Positive Pairs: Scalable Augmentation with Query Extraction and Generation

Rui Meng, Ye Liu, Semih Yavuz, Divyansh Agarwal, Lifu Tu, Ning Yu, Jianguo Zhang, Meghana Bhat, Yingbo Zhou

arXiv 2022

pdf



Membership Inference Attacks Against Text-to-image Generation Models

Yixin Wu, Ning Yu, Zheng Li, Michael Backes, Yang Zhang

arXiv 2022

pdf



Auditing Membership Leakages of Multi-Exit Networks

Zheng Li, Yiyong Liu, Xinlei He, Ning Yu, Michael Backes, Yang Zhang

CCS 2022

pdf code slides



RepMix: Representation Mixing for Robust Attribution of Synthesized Images

Tu Bui, Ning Yu, John Collomosse

ECCV 2022

pdf code poster video



Deep Learning Accurately Predicts Food Categories and Nutrients Based on Ingredient Statements

Peihua Ma, Zhikun Zhang, Ying Li, Ning Yu, Jiping Sheng, Hande Küçük McGinty, Qin Wang, Jaspreet Ahuja

Food Chemistry 2022

pdf



Responsible Disclosure of Generative Models Using Scalable Fingerprinting

Ning Yu*, Vladislav Skripniuk*, Dingfan Chen, Larry Davis, Mario Fritz

ICLR 2022 Spotlight

pdf project code poster video press



RelaxLoss: Defending Membership Inference Attacks without Losing Utility

Dingfan Chen, Ning Yu, Mario Fritz

ICLR 2022 Spotlight

pdf code poster video press



Human-Centric Deep Generative Models: The Blessing and The Curse

Ning Yu

Ph.D. Dissertation 2021, University of Maryland and Max Planck Institute for Informatics

pdf



Deep Video Inpainting Detection

Peng Zhou, Ning Yu, Zuxuan Wu, Larry Davis, Abhinav Shrivastava, Ser-Nam Lim

BMVC 2021

pdf code video



Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data

Ning Yu*, Vladislav Skripniuk*, Sahar Abdelnabi, Mario Fritz

ICCV 2021 Oral

pdf project code poster video press



Dual Contrastive Loss and Attention for GANs

Ning Yu, Guilin Liu, Aysegul Dundar, Andrew Tao, Bryan Catanzaro, Larry Davis, Mario Fritz

ICCV 2021

pdf project code poster video press



Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis

Yang He, Ning Yu, Margret Keuper, Mario Fritz

IJCAI 2021

pdf project code video press



Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Hui-Po Wang, Ning Yu, Mario Fritz

CVPR 2021

pdf project code poster video press



Application of Deep Learning for Image-based Chinese Market Food Nutrients Estimation

Peihua Ma, Chun Pong Lau, Ning Yu, An Li, Jiping Sheng

Food Chemistry 2021

pdf



Image-based Nutrient Estimation for Chinese Dishes Using Deep Learning

Peihua Ma, Chun Pong Lau, Ning Yu, An Li, Ping Liu, Qin Wang, Jiping Sheng

Food Research International 2021

pdf press



Application of Machine Learning for Predicting Label Nutrients Using USDA Global Branded Food Products Database (BFPD)

Peihua Ma, An Li, Ning Yu, Ying Li, Rahul Bahadur, Qin Wang, Jaspreet Ahuja

Journal of Food Composition and Analysis 2021

pdf press



Long-Tailed Recognition Using Class-Balanced Experts

Saurabh Sharma, Ning Yu, Mario Fritz, Bernt Schiele

GCPR 2020

pdf code video



GAN-Leaks: A Taxonomy of Membership Inference Attacks against GANs

Dingfan Chen, Ning Yu, Yang Zhang, Mario Fritz

CCS 2020

pdf code poster video (short) video (full)



Inclusive GAN: Improving Data and Minority Coverage in Generative Models

Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, Mario Fritz

ECCV 2020

pdf project code video (short) video (full) press



Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints

Ning Yu, Larry Davis, Mario Fritz

ICCV 2019

pdf project code poster press



Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture

Ning Yu, Connelly Barnes, Eli Shechtman, Sohrab Amirghodsi, Michal Lukáč

CVPR 2019

pdf project code poster press



Learning to Detect Multiple Photographic Defects

Ning Yu, Xiaohui Shen, Zhe Lin, Radomír Měch, Connelly Barnes

WACV 2018

pdf supp project code poster slides



Supervoxel-Based Hierarchical Markov Random Field Framework for Multi-Atlas Segmentation

Ning Yu, Hongzhi Wang, Paul Yushkevich

MICCAI Workshop on Patch MI 2016

pdf project slides



A Superpixel-Based Framework for Automatic Tumor Segmentation on Breast DCE-MRI

Ning Yu, Jia Wu, Susan Weinstein, Bilwaj Gaonkar, Brad Keller, Ahmed Ashraf, YunQing Jiang, Christos Davatzikos, Emily Conant, Despina Kontos

SPIE Medical Imaging 2015 Oral, best student paper finalist

pdf project slides press



Quantification of Tumor Changes during Neoadjuvant Chemotherapy with Longitudinal Breast DCE-MRI Registration

Jia Wu, Yangming Ou, Susan Weinstein, Emily Conant, Ning Yu, Vahid Hoshmand, Brad Keller, Ahmed Ashraf, Mark Rosen, Angela DeMichele, Christos Davatzikos, Despina Kontos

SPIE Medical Imaging 2015

pdf



Tumor Heterogeneity Patterns of DCE-MRI Parametric Response Maps May Augment Early Assessment of Neoadjuvant Chemotherapy: A Pilot Study of ACRIN 6657/I-SPY 1

Jia Wu, Susan Weinstein, Andrew Oustimov, Lauren Pantalone, Ning Yu, Yangming Ou, Mark Rosen, Angela DeMichele, Christos Davatzikos, Despina Kontos

RSNA 2015

pdf



A Feasibility Study Investigating the Use of Quantitative Measures of Spatio-Temporal Tumor Heterogeneity Derived from 4D Breast DCE-MRI Registration as a Biomarker of Response to Neoadjuvant Chemotherapy

Jia Wu, Yangming Ou, Susan Weinstein, Emily Conant, Ning Yu, Vahid Hoshmand, Brad Keller, Ahmed Ashraf, Mark Rosen, Angela DeMichele, Christos Davatzikos, Despina Kontos

ISMRM 2014

pdf



Robust Feature Points Correspondences for Visual Object Tracking

Ning Yu

Undergraduate Thesis 2013, Huazhong University of Science and Technology

pdf project slides



Patents


Systems And Methods For Controllable Data Generation From Text
Shiyu Wang, Yihao Feng, Tian Lan, Ning Yu, Yu Bai, Ran Xu, Huan Wang, Caiming Xiong, Silvio Savarese
US Application No. 18,423,081

Systems and Methods for Zero-shot Cross-modal Reasoning via Uni-modal Projections to Large Language Models
Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Silvio Savarese, Shafiq Joty, Ran Xu, Caiming Xiong, Juan Carlos Niebles
US Application No. 18,400,477

Systems and Methods for Multimodal Pretraining for Three-Dimensional Models
Le Xue, Ning Yu, Shu Zhang, Junnan Li, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese
US Application No. 18,493,035

Systems and Methods for Controllable Image Generation
Ning Yu, Can Qin, Shu Zhang, Yihao Feng, Xinyi Yang, Ran Xu
US Application No. 18,477,764

Systems and Methods for Feedback Based Instructional Visual Editing
Shu Zhang, Xinyi Yang, Yihao Feng, Chia-Chih Chen, Ning Yu, Ran Xu
US Application No. 18,350,876

Systems and Methods for Unsupervised Training in Text Retrieval Tasks
Rui Meng, Yingbo Zhou, Ye Liu, Semih Yavuz, Ning Yu
US Application No. 18,303,313

Systems and Methods for Multimodal Layout Designs of Digital Publications
Ning Yu, Chia-Chih Chen, Zeyuan Chen, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Rui Meng
US Application No. 18,161,680

Systems and Methods for Text-to-Image Generation Using Language Models
Ning Yu, Can Qin, Chen Xing, Shu Zhang, Stefano Ermon, Caiming Xiong, Ran Xu
US Application No. 18,162,535

Systems and Methods for Open-Vocabulary Instance Segmentation
Ning Yu, Vibashan VS, Chen Xing, Juan Carlos Niebles, Ran Xu
US Application No. 18,159,318

Systems and Methods for Attention Mechanism in Three-Dimensional Object Detection
Manli Shu, Le Xue, Ning Yu, Roberto Martín-Martín, Juan Carlos Niebles, Caiming Xiong, Ran Xu
US Application No. 18,161,661

Neural Network Training Technique
Guilin Liu, Ning Yu, Aysegul Dundar, Andrew Tao, Bryan Catanzaro
US Application No. 17,165,745

Texture Interpolation Using Neural Networks
Connelly Barnes, Sohrab Amirghodsi, Michal Lukáč, Eli Shechtman, Ning Yu
US Patent No. 10,818,043
pdf

Digital Image Defect Identification and Correction
Radomír Měch, Ning Yu, Xiaohui Shen, Zhe Lin
US Patent No. 10,810,721
pdf

Selected Awards


[2022] ICML'22 Top Reviewer
[2021] Twitch (Amazon) Research Fellowship
[2020] Qualcomm Innovation Fellowship Finalist
[2019] Qualcomm Innovation Fellowship Finalist
[2015] SPIE Best Student Paper Finalist
[2015] SPIE Travel Scholarship
[2012] Chinese National Fellowship
[2012] Microsoft Young Fellowship
[2012] Meritorious Winner of the Mathematical Contest in Modeling
[2011] 2nd Prize of the Chinese Mathematical Contest in Modeling

Mentoring


Ryan Burgert @StonyBrook
Yimu Wang @UWaterloo
Dipanwita Guhathakurta @GaTech
Yuancheng Xu @UMD
Bahri Batuhan Bilecen @Bilkent
Yijiang Li @JHU
Kai Wang @GaTech
Shiyu Wang @Emory
Artemis Panagopoulou @UPenn
You-Ming Chang @NCTU
Chen Yeh @NCTU
Qian Wang @HUST
Minxing Zhang @CISPA
Manli Shu @UMD
Vibashan VS @JHU
Can Qin @NEU
Yuan Xin @CISPA
Yixin Wu @CISPA
Shuo Wen @MPI-INF
Zeyang Sha @CISPA
Yihan Ma @CISPA
Jitesh Jain @IIT
Zheng Li @CISPA
Vladislav Skripniuk @Audatic
Hui-Po Wang @CISPA
Dingfan Chen @CISPA
Saurabh Sharma @UCSB

Chairing


CVPR area chair 2023,2024
ICCV area chair 2023
ECCV area chair 2024
NeurIPS area chair 2023
ICLR area chair 2024
ICML session chair 2022
AAAI senior program committee 2023,2024
AISTATS area chair 2023,2024
BMVC area chair 2022,2023
WACV area chair 2023

Reviewing


CVPR since 2020
ICCV since 2019
ECCV since 2020
NeurIPS since 2021
ICML since 2021
ICLR since 2022
AAAI since 2020
IJCAI since 2020
SIGGRAPH since 2021
Eurographics since 2020
ICRA since 2020
TPAMI since 2019
IJCV since 2021
TOG since 2020
PR Letters since 2020
Neurocomputing since 2019
TVCJ since 2018

Teaching


@University of Virginia
Computer Graphics
Operating Systems
Information Technology

@University of Pennsylvania
Probability Theory