Yapeng Tian

Assistant Professor, The University of Texas at Dallas

yapeng.tian@utdallas.edu

ECSS 4.211

Bio

I am an Assistant Professor in the Computer Science Department of UT Dallas and lead the Computer Vision and Multimodal Computing (CVMC) Lab. Before coming to UTD, I finished my PhD at University of Rochester, advised by Chenliang Xu, my master degree at Tsinghua University working with Wenming Yang, and B.E degree at Xidian University. I was a visiting student at SIAT advised by Yu Qiao. I did internships at Adobe Research with Dingzeyu Li and Meta with Alexander Richard. I am interested in solving core computer vision, computer audition, and machine learning problems and applying the developed learning approaches to broad AI applications, such as multisensory perception, computational photography, AR/VR, accessibility, and healthcare. My work has been recognized with awards including the AAAI New Faculty Highlights, Cisco Faculty Research Award, and Amazon Research Award.

ICCV 2025 CV4A11y Workshop Announcement:
 We are excited to announce the CV4A11y workshop on Vision Foundation Models and Generative AI for Accessibility: Challenges and Opportunities at ICCV 2025. We will be inviting paper submissions—details coming soon. Stay tuned!

Research Highlights

    Audio-Visual Scene Understanding: To achieve truly intelligent systems, we must move beyond scene understanding that relies solely on individual senses like sight or sound. Our research pioneers a multimodal approach, integrating computer vision and audition to unlock the rich information available through combined audio-visual perception. We tackle fundamental challenges in this emerging field, developing unified, explainable, and robust multimodal models for video understanding.

    Audio-Visual Scene Generation: We're pushing beyond scene perception to content creation, developing audio-visual intelligent systems that can generate realistic sounds and videos from diverse multimodal inputs. Our work includes joint audio-visual generation, spatial audio generation, and video and text guided audio generation models.

    AI for Accessibility and Healthcare: We develop computer vision and AI models to empower individuals with disabilities and improve healthcare. Our work spans: (a) Assistive Technologies: Creating systems to aid those with autism, blind or low vision, and hearing impairments. (b) Medical Imaging: Developing AI to enhance medical images for improved diagnosis and treatment.

    Image and Video Processing: I enjoy building computer vision algorithms to improve image and video quality in both automated and creative ways. My research involves developing a range of image and video restoration models capable of generating photorealistic outputs. This work has also been applied to medical imaging, including MRI data.

News

  • 04/2025: I will be serving as an Area Chair for NeurIPS 2025.
  • 04/2025: We are excited to announce the CV4A11y workshop at ICCV 2025. More details coming soon. Stay tuned!
  • 04/2025: Guest lecture at Texas A&M.
  • 02/2025: Two papers accepted at CVPR 2025.
  • 02/2025: Our Multimodal Large Language Model Pruning paper accepted at PAKDD 2025.
  • 01/2025: Invited talk at UNT Artificial Intelligence (AI) Seminar.
  • 12/2024: Our spatial audio generation paper accepted at ICASSP 2025.
  • 12/2024: emoji_eventsDAVIS won the ACCV Best Paper Award, Honorable Mention!
  • 11/2024: One paper accepted at WACV 2025 and one journal article accepted by IEEE TPAMI.
  • 10/2024: Our Audio-Visual Dataset Distillation paper accepted at TMLR journal.
  • 10/2024: I will be serving as an Area Chair for CVPR 2025.
  • 10/2024: emoji_eventsARSports won IEEE ISMAR'24 IDEATExR workshop Best Paper Award!
  • 10/2024: Invited talk at UTSW.
  • 09/2024: Our Continual Audio-Visual Sound Separation paper accepted at NeurIPS 2024.
  • 09/2024: emoji_eventsCookAR received UIST'24 Belonging & Inclusion Best Paper Award!
  • 09/2024: Received an NIH R01 grant! This exciting project will focus on AI/AR-assisted Vision for people with low vision. We're excited to collaborate with Prof. Yuhang Zhao at UW-Madison (lead institute) and Prof. Jon E. Froehlich at UW. Check out our preliminary work: CookAR.
  • 09/2024: Our audio-visual question answering paper accepted at EMNLP 2024.
  • 09/2024: Two papers accepted at ACCV 2024. One on audio-visual sound separation and another on language-guided audio-visual editing.
  • 09/2024: Our Audio-Visual Autism Behavior Recognition paper acceted at IEEE TMM.
  • 08/2024: I will be serving as an Area Chair for ICLR 2025 and a SPC for AAAI 2025.
  • 08/2024: William received the Jonsson School of Engineering and Computer Science Award for his Undergraduate Research Project!
  • 07/2024 Our CookAR paper accepted at UIST 2024.
  • 07/2024: We will be organizing an Audio Imagination workshop at NeurIPS 2024. More details coming soon!
  • 07/2024: Our mentored high school students received the Best Science Education Award at the 2024 CAST-STEM Bridge Summer Camp.
  • 07/2024: One paper accepted at BMVC 2024.
  • 06/2024: Our EgoVSR paper accepted at IEEE TCSVT.
  • 05/2024: One paper accepted at ACM TOMM.
  • 05/2024: One paper accepted at ACM C&C.
  • 05/2024: Seven papers accepted at CVPR Workshops.
  • 04/2024: Dr. Rollins and I received a UTD SPIRe grant.
  • 04/2024: Received an Amazon Research Award.
  • 03/2024: Congrats to Zeke Barnett, a K12 student in the lab! He will be joining CMU for his undergraduate study.
  • 03/2024: One paper accepted at NAACL 2024.
  • 03/2034: One journal article accepted at Medical Image Analysis.
  • 03/2024: One journal article accepted at IEEE TMM.
  • 03/2024: Received UTD Undergraduate Research Apprenticeship Program (URAP) award.
  • 02/2024: We are organizing an ELVM: Efficient Large Vision Models workshop at CVPR 2024.
  • 02/2024: One paper accepted at CVPR 2024.
  • 02/2024: One paper accepted at CHI 2024.
  • 10/2023: Invited talk at UTD-DFWCSTA Battle of the Brains - Conference & Contest for K12 students.
  • 10/2023: Invited lightning talk at Workshop on Imaging and Data Science.
  • 10/2023: One paper accepted at WACV 2024.
  • 10/2023: Listed in 2022 World's Top 2% Scientists by Stanford University.
  • 10/2023: Invited talk at Do Good with Data Webinar for K12 students.
  • 09/2023: Two papers accepted at NeurIPS 2023.
  • 09/2023: One paper accepted at UIST 2023.
  • 09/2023: Five papers accepted at ICCV AV4D workshop.
  • 07/2023: Three papers accepted at ICCV 2023.
  • 06/2023: One paper accepted at MICCAI 2023. We are organizing a Cardiac MRI Reconstruction Challenge in conjunction with MICCAI 2023.
  • 06/2023: Invited talk at Sight and Sound Workshop @ CVPR 2023.
  • 06/2023: I will serve as a SPC for AAAI 2024.
  • 06/2023: Received an Adobe Research Gift.
  • 06/2023: One journal paper accepted at IEEE Transactions on Image Processing.
  • 05/2023: Three papers accepted at CVPR Sight and Sound Workshop.
  • 03/2023: I will serve as an Execution Area Chair for VALSE.
  • 03/2023: Received an inaugural Undergraduate Research Apprenticeship Program (URAP) award.
  • 03/2023: Received a Cisco Faculty Research Award.
  • 03/2023: I will be co-organizing a Cardiac MRI Reconstruction Challenge in conjunction with MICCAI 2023.
  • 02/2023: Three papers accepted at CVPR 2023.
  • 02/2023: Please check out our new AV-NeRF paper. In this work, we pose and tackle a Real-World Audio-Visual Scene Synthesis problem.
  • 02/2023: One journal paper accepted at IEEE Signal Processing Letters.
  • 02/2023: One journal paper accepted at IEEE Transactions on Neural Networks and Learning Systems.
  • 01/2023: Two papers accepted at ICLR 2023.
  • 11/2022: Selected for the 2023 AAAI New Faculty Highlights Program.
  • 10/2022: Invited talk at AV4D Workshop @ ECCV 2022.
  • 09/2022: One paper accepted at NeurIPS 2022. Congratulations to Shentong!
  • 09/2022: Two papers accepted at ECCV@AV4D 2022 .
  • 08/2022: Please check out our new article "Learning in Audio-visual Context: A Review, Analysis, and New Perspective."
  • 08/2022: I start as an assistant professor in CS at UTD.
  • 07/2022: I will serve as a Senior Program Committee (SPC) Member for AAAI 2023.
  • 07/2022: One paper accepted at ECCV 2022.
  • 06/2022: One paper accepted at MICCAI 2022.
  • 06/2022: Successfully defended my dissertation! Thanks to everyone who supported me and helped me along the way.
  • 04/2022: I will attend CVPR'22 Doctoral Consortium.
  • 03/2022: Two works: audio-visual question answering and MRI SR are accepted by CVPR 2022.
  • 12/2021: Two papers are accepted by AAAI 2022.
  • 10/2021: One paper on sounding object localization is accepted by BMVC 2021!
  • 07/2021: One paper on video matting is accepted by ICCV 2021!
  • 03/2021: Our two works: co-learn sounding object visual grounding and sound separation and audio-visual robustness are accepted by CVPR 2021!
  • 02/2021: We will co-organize a CVPR 2021 Tutorial on Audio-visual Scene Understanding!
  • 01/2021: Co-organized the WACV 2021 Tutorial on Audio-visual Scene Understanding. More details can be found in our website.
  • 10/2020: I was in the top 10% of high-scoring reviewers for NeurIPS 2020!
  • 07/2020: Our audio-visual video parsing work got accepted by ECCV 2020 as a Spotlight.
  • 05/2020: Our three papers will be presented in the CVPR 2020 Sight and Sound workshop.
  • 02/2020: Two papers on video restoration got accepted by CVPR 2020! Congratulations to all co-authors!
  • 01/2020: RDN is accepted by IEEE TPAMI! Congratulations to Yulun!
  • 12/2019: Please check our deep audio prior paper.
  • 08/2019: One paper is accepted by IEEE TIP. Congratulations to Xuechen!
  • 07/2019: One paper is accepted by ICCV 2019. Congratulations to Wei!
  • 05/2019: Our two works: audio-visual event localization and audio-visual video captioning will be presented in the CVPR 2019 Sight and Sound workshop.
  • 02/2019: I will serve as an ICCV 2019 reviewer.
  • 12/2018: Two papers are posted on ArXiv. Please watch the corresponding demos.
  • 07/2018: One paper is accepted by ECCV 2018! AVE dataset and codes have been released.
  • 02/2018: One paper is accepted by CVPR 2018. Congratulations to Yulun!
  • 07/2017: I recieve 'Outstanding Graduate of Tsinghua university' and 'Outstanding Master Thesis Award'.
  • 03/2017: I will join Prof. Chenliang Xu's lab to pursue a PhD degree at University of Rochester!

Students

Students at UTD:
Siva Sai Nagender Vasireddy (PhD student; Fall 2022)
Shijian Deng (PhD student; Spring 2023)
Saksham Singh Kushwaha (PhD student; Summer 2023)
Jia Li (PhD student; Fall 2024)
Xinpeng Li (PhD student; Fall 2024)

Collaborated External Students:
Tianyu Yang (PhD student at University of Notre Dame)
Shentong Mo (PhD student at Carnegie Mellon University)
Kai Wang (PhD student at University of Toronto)

Alumni:
Ziru Huang (Visiting student; Tsinghua University; 2024)
Yiyang Nan (Graduate student at Brown University; Spring 2023-2024; Next: researcher at Cohere for AI)
Matthew Wang (K12; Summer 2023-2024)
William Doan (Undergraduate; Fall 2023 - Summer 2024; Jonsson School of Engineering and Computer Science Award)
Zeke Barnett (K12; Parish Episcopal School at Dallas, Spring 2023 - Spring 2024; Next Undergraduate at CMU)
Anikait Bharadwaj (K12; Frisco ISD; Spring 2024)
Aditya Kulkarni (Undergraduate; Spring 2023; Next Meta)
Michael Yang (K12; Summer 2023)
Atmin Mehul Sheth (Undergraduate at UTD; 2023)
Yuxin Ye (Graduate student at Tsinghua University)
Yichen Chi (Graduate student at Tsinghua University)
Junhao Gu (PhD student at Tsinghua University)
Jiamiao Zhang (Graduate student at Tsinghua University)
Sen Fang (Undergraduate at Victoria University, Next: PhD student at Rutgers University)
Sasha Kaplan (Undergraduate; Spring 2023)
Sisi Aarukapalli (Undergraduate; Summer 2023)
Harsh Singh (PhD student at UTD; Spring and Summer 2023; Next: CV MSC at MBZUAI)
Yulang Wu (Graduate student at UTD CS, Spring 2023; Next: Postdoc at University of California San Francisco)
Guangyao Li (PhD student at Renmin University of China, Fall 2020 - Spring 2023)
Shijian Deng (Graduate student at University of Rochester; next: PhD student at UTD)
Hai Wang (Graduate student at Tsinghua University; next: PhD student at UCL)
Sizhe Li (Undergraduate student at University of Rochester; next: Visiting student at MIT)
Yiyang Su (Undergraduate at University of Rochester; next: PhD student at Michigan State University)
Rohan Sharma (Graduate student at University of Rochester; next: PhD student at SUNY Buffalo)
Chenxiao Guan (Undergraduate at University of Rochester; next: Graduate student at CMU)

Publications

Most recent publications on Google Scholar.
‡ indicates equal contribution.

  • All
  • Preprint
  • Vision+Sound
  • Video Restoration
  • Image Restoration

Towards Online Multi-Modal Social Interaction Understanding

Xinpeng Li, Shijian Deng, Bolin Lai, Weiguo Pian, James M. Rehg, Yapeng Tian

Preprint'25.

Paper

Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?

Jia Li, Wenjie Zhao, Ziru Huang, Yunhui Guo, Yapeng Tian

Preprint'25.

Paper

Modality-Inconsistent Continual Learning of Multimodal Large Language Models

Weiguo Pian, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

Preprint'24.

Paper

Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach

Shijian Deng, Wentian Zhao, Yu-Jhe Li, Kun Wan, Daniel Miranda, Ajinkya Kale, Yapeng Tian

Preprint'24.

Paper

AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation

Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, Yapeng Tian

Preprint'24.

Paper

Text-to-Audio Generation Synchronized with Videos

Shentong Mo, Jing Shi, Yapeng Tian

Preprint'24.

Paper

Robust Active Speaker Detection in Noisy Environments

Siva Sai Nagender Vasireddy, Chenxu Zhang, Xiaohu Guo, Yapeng Tian

Preprint'24.

Paper

SignDiff: Learning Diffusion Models for American Sign Language Production

Sen Fang, Chunyu Sui, Xuedong Zhang, Yapeng Tian

Preprint'23.

Paper

Learning in Audio-visual Context: A Review, Analysis, and New Perspective

Yake Wei, Di Hu, Yapeng Tian, Xuelong Li

Preprint'22.

Paper Project

VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation

Saksham Singh Kushwaha, Yapeng Tian

CVPR'25: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Project

Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level

Andong Deng, Tongjia Chen, Shoubin Yu, Taojiannan Yang, Lincoln Spencer, Yapeng Tian, Ajmal Saeed Mian, Mohit Bansal, Chen Chen

CVPR'25: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Project

Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models

Saksham Singh Kushwaha, Jianbo Ma, Mark R. P. Thomas, Yapeng Tian, and Avery Bruni

ICASSP'25: IEEE International Conference on Acoustics, Speech, and Signal Processing.

Paper

Language-Guided Adaptive Vision Token Pruning for Efficient Multimodal Large Language Models

Omer Faruk Deniz, Tarik Arici, Fatemeh Sheikholeslami, Burak Gozluklu, Ameni Trabelsi, Suleiman Khan, Yapeng Tian, Latifur Khan

PAKDD'25 Oral: The Pacific-Asia Conference on Knowledge Discovery and Data Mining.

Paper

Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters

Steven Hogue, Chenxu Zhang, Yapeng Tian, Xiaohu Guo

WACV'25: IEEE/CVF Winter Conference on Applications of Computer Vision.

Paper

DiffIR: Efficient Diffusion Model for Image Restoration

Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, Radu Timotfe, Luc Van Gool

TPAMI'25: IEEE Transactions on Pattern Analysis and Machine Intelligence.

Paper Code

Audio-Visual Dataset Distillation

Saksham Singh Kushwaha, Siva Sai Nagender Vasireddy, Kai Wang, Yapeng Tian

TMLR'24: Transactions on Machine Learning Research

Paper

Continual Audio-Visual Sound Separation

Weiguo Pian, Yiyang Nan, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

NeurIPS'24: The Annual Conference on Neural Information Processing Systems

Paper

Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition

Shijian Deng, Erin Kosloski, Siddhi Patel, Zeke A Barnett, Yiyang Nan, Alexander M Kaplan, Sisira Aarukapalli, William Doan, Matthew Wang, Harsh Singh, Rollins Pamela, Yapeng Tian

TMM'24: IEEE Transactions on Multimedia.

Paper Project

emoji_eventsTowards AI-Powered AR for Enhancing Sports Playability for People with Low Vision: An Exploration of ARSports (Best Paper Award)

Jaewook Lee, Yang Li, Dylan Bunarto, Eujean Lee, Olivia Wang, Adrian Rodriguez, Yuhang Zhao, Yapeng Tian, Jon E. Froehlich

ISMAR IDEATExR'24 : International Symposium on Mixed and Augmented Reality Workshop.

Paper

emoji_events CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low Vision (Belonging & Inclusion Best Paper Award)

Jaewook Lee, Andrew D. Tjahjadi, Jiho Kim, Junpu Yu, Minji Park, Jiawen Zhang, Jon E. Froehlich, Yapeng Tian, Yuhang Zhao

UIST'24: ACM Symposium on User Interface Software and Technology.

Paper Project

emoji_events DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models
(Best Paper Honorable Mention)

Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, Chenliang Xu

ACCV'24 Oral: Asian Conference on Computer Vision.

Paper

Language-Guided Joint Audio-Visual Editing Via One-Shot Adaptation

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

ACCV'24: Asian Conference on Computer Vision.

Paper

SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering

Tianyu Yang, Yiyang Nan, Lisen Dai, Zhenwen Liang, Yapeng Tian, Xiangliang Zhang

EMNLP'24: Empirical Methods in Natural Language Processing (Findings)

Paper

Towards Long Form Audio-visual Video Understanding

Wenxuan Hou, Guangyao Li, Yapeng Tian, Di Hu

TOMM'24: ACM Trans. on Multimedia Computing, Communications and App.

Paper Project

Benchmarking and Optimizing Federated Learning with Hardware-related Metrics

Kai Pan, Yapeng Tian, Yinhe Han, Yiming Gan

BMVC'24: British Machine Vision Conference

Paper

EgoVSR: Towards High-Quality Egocentric Video Super-Resolution

Yichen Chi, Junhao Gu, Jiamiao Zhang, Wenming Yang, Yapeng Tian

TCSVT'24: IEEE Transactions on Circuits and Systems for Video Technology.

Paper

MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos

Zheng Ning, Zheng Zhang, Jerrick Ban, Kaiwen Jiang, Ruohong Gan, Yapeng Tian, Toby Jia-Jun Li

C&C'24: ACM Conference on Creativity & Cognition.

Paper

AV-Mamba: Cross-Modality Selective State Space Models for Audio-Visual Question Answering

Ziru Huang, Jia Li, Wenjie Zhao, Yunhui Guo, Yapeng Tian

CVPRW'24: CVPR Signt and Sound Workshop

Paper

Learning Continual Audio-Visual Sound Separation Models

Weiguo Pian, Yiyang Nan, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

CVPRW'24: CVPR Signt and Sound Workshop

Paper

Audio-Visual Autism Behavior Recognition with MMLMs

Shijian Deng, Erin Kosloski, Siddhi Patel, Zeke A Barnett, Yiyang Nan, Alexander M Kaplan, Sisira Aarukapalli, William Doan, Matthew Wang, Harsh Singh, Rollins Pamela, Yapeng Tian

CVPRW'24: CVPR Signt and Sound Workshop

Paper

Dataset distillation for audio-visual datasets

Saksham Singh Kushwaha, Siva Sai Nagender Vasireddy, Kai Wang, Yapeng Tian

CVPRW'24: CVPR Signt and Sound Workshop

Paper

DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures

Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, Xiaohu Guo

CVPRW'24: CVPR HuMoGen Workshop

Paper

Towards Efficient Audio-Visual Learners via Empowering Pre-trained Vision Transformers with Cross-Modal Adaptation

Kai Wang, Yapeng Tian, Dimitrios Hatzinakos

CVPRW'24: CVPR Multimodal Foundation Models Workshop

Paper Code

MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers

Tanvir Mahmud, Shentong Mo, Yapeng Tian, Diana Marculescu

CVPRW'24: CVPR Efficient Deep Learning for Computer Vision Workshop

Paper

T-VSL: Text-Guided Visual Sound Source Localization in Mixtures

Tanvir Mahmud, Yapeng Tian, Diana Marculescu

CVPR'24: IEEE/CVF Conference on Computer Vision and Pattern Recognition

Paper Code

OSCaR: Object State Captioning and State Change Representation

Nguyen Nguyen, Jing Bi, Ali Vosoughi, Yapeng Tian, Pooyan Fazli, Chenliang Xu

NAACL'24: The North American Chapter of the Association for Computational Linguistics (Findings)

Paper

SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers

Zheng Ning, Brianna Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, Toby Li

CHI'24: The ACM Conference on Human Factors in Computing Systems.

Paper

STADNet: Spatial-Temporal Attention-Guided Dual-Path Network for cardiac cine MRI super-resolution

Jun Lyu, Shuo Wang, Yapeng Tian, Jing Zou, Shunjie Dong, Chengyan Wang, Angelica I Aviles-Rivero, Jing Qin

MIA'24: Medical Image Analysis

Paper

Unveiling cross modality bias in visual question answering: A causal view with possible worlds vqa

Ali Vosoughi‡, Shijian Deng‡, Songyang Zhang, Yapeng Tian, Chenliang Xu, Jiebo Luo

TMM'24: IEEE Transactions on Multimedia

Paper

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Yuxin Ye, Wenming Yang, Yapeng Tian

WACV'24: Winter Conference on Applications of Computer Vision.

Paper Project

Disentangled counterfactual learning for physical audiovisual commonsense reasoning

Changsheng Lv, Shuai Zhang, Yapeng Tian, Mengshi Qi, Huadong Ma

NeurIPS'23: The Annual Conference on Neural Information Processing Systems.

Paper Code

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

NeurIPS'23: The Annual Conference on Neural Information Processing Systems.

Paper Project

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data

Zheng Zhang‡, Zheng Ning‡, Chenliang Xu Yapeng Tian, Toby Jia-Jun Li

UIST'23: ACM Symposium on User Interface Software and Technology.

Paper

Towards Robust Active Speaker Detection

Siva Sai Nagender Vasireddy, Chenxu Zhang, Xiaohu Guo, Yapeng Tian

ICCVW'23: ICCV AV4D Workshop .

Paper

Position-Aware Audio-Visual Separation for Spatial Audio

Yuxin Ye, Wenming Yang, Yapeng Tian

ICCVW'23: ICCV AV4D Workshop .

Paper

Towards Better Egocentric Action Understanding in a Multi-Input Multi-Output View

Wenxuan Hou, Ruoxuan Feng, Yixin Xu, Yapeng Tian, Di Hu

ICCVW'23: ICCV AV4D Workshop .

Paper

Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

ICCVW'23: ICCV AV4D Workshop .

Paper

Separating Invisible Sounds Toward Universal Audio-Visual Scene-Aware Sound Separation

Yiyang Su, Ali Vosoughi, Shijian Deng, Yapeng Tian, Chenliang Xu

ICCVW'23: ICCV AV4D Workshop .

Paper

Audio-Visual Class-Incremental Learning

Weiguo Pian‡, Shentong Mo‡, Yunhui Guo, Yapeng Tian

ICCV'23: IEEE/CVF International Conference on Computer Vision.

Paper Code

Class-Incremental Grouping Network for Continual Audio-Visual Learning

Shentong Mo‡, Weiguo Pian‡, Yapeng Tian

ICCV'23: IEEE/CVF International Conference on Computer Vision.

Paper Code

DiffIR: Efficient Diffusion Model for Image Restoration

Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, Luc Van Gool

ICCV'23: IEEE/CVF International Conference on Computer Vision.

Paper Code

Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI

Jiamiao Zhang, Yichen Chi, Jun Lyu, Wenming Yang, Yapeng Tian

MICCAI'23: Medical Image Computing and Computer-Assisted Intervention.

Paper Code

Meta-Learning based Degradation Representation for Blind Super-Resolution

Bin Xia, Yapeng Tian, Yulun Zhang, Yucheng Hang, Wenming Yang, Qingmin Liao

TIP'23: IEEE Transactions on Image Processing.

Paper Code

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Shentong Mo, Yapeng Tian

CVPRW'23: CVPR Sight and Sound Workshop.

Paper

DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment

Shentong Mo, Jing Shi, Yapeng Tian

CVPRW'23: CVPR Sight and Sound Workshop.

Paper

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

CVPRW'23: CVPR Sight and Sound Workshop.

Paper Project

Audio-Visual Grouping Network for Sound Localization from Mixtures

Shentong Mo, Yapeng Tian

CVPR'23: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Egocentric Audio-Visual Object Localization

Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu

CVPR'23: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Structured Sparsity Learning for Efficient Video Super-Resolution

Bin Xia, Jingwen He, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Luc Van Gool

CVPR'23: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Knowledge Distillation based Degradation Estimation for Blind Super-Resolution

Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, Luc Van Gool

ICLR'23: International Conference on Learning Representations.

Paper Code

Basic Binary Convolution Unit for Binarized Image Restoration Network

Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, Luc Van Gool

ICLR'23: International Conference on Learning Representations.

Paper Code

Stdan: deformable attention network for space-time video super-resolution

Hai Wang, Xiaoyu Xiang, Yapeng Tian, Wenming Yang, Qingmin Liao

TNNLS'23: IEEE Transactions on Neural Networks and Learning Systems.

Paper Code

GDSSR: Toward Real-World Ultra-High-Resolution Image Super-Resolution

Yichen Chi, Wenming Yang, Yapeng Tian

SPL'23: IEEE Signal Processing Letters.

Paper

Towards Unified, Explainable, and Robust Multisensory Perception

Yapeng Tian

AAAI'23: AAAI Conference on Artificial Intelligence. (NFH program)

Paper

Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing

Shentong Mo, Yapeng Tian

NeurIPS'22: The Annual Conference on Neural Information Processing Systems.

Paper Code

Learning Spatio-Temporal Downsampling for Effective Video Upscaling

Xiaoyu Xiang, Yapeng Tian, Vijay Rengarajan, Lucas Young, Bo Zhu, Rakesh Ranjan

ECCV'22: European Conference on Computer Vision.

Paper

Audio-Visual Scene Understanding Towards Unified, Explainable, and Robust Multisensory Perception

Yapeng Tian

PhD Thesis

Paper

DuDoCAF: Dual-Domain Cross-Attention Fusion with Recurrent Transformer for Fast Multi-contrast MR Imaging

Jun Lyu, Bin Sui, Chengyan Wang, Yapeng Tian, Qi Dou, and Jing Qin

MICCAI'22: Medical Image Computing and Computer Assisted Intervention.

Paper

Audio-Visual Object Localization in Egocentric Videos

Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu

CVPRW'22: CVPR Workshops

Egocentric audio-visual learning.

Paper

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

Guangyao Li‡, Yake Wei‡, Yapeng Tian‡, Chenliang Xu, Ji-Rong Wen, and Di Hu

CVPR'22 Oral: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Video Code Project

Transformer-empowered Multi-contrast MRI Super-Resolution

Guangyuan Li, Jun Lv, Yapeng Tian, Qi Dou, Chengyan Wang, Chenliang Xu, Jing Qin

CVPR'22: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-based Super-Resolution

Bin Xia, Yapeng Tian, Yucheng Hang, Wenming Yang, Qingmin Liao, Jie Zhou

AAAI'22: The AAAI Conference on Artificial Intelligence.

Paper Code

Efficient Non-Local Contrastive Attention for Image Super-Resolution

Bin Xia‡, Yucheng Hang‡, Yapeng Tian, Wenming Yang, Qingmin Liao, Jie Zhou

AAAI'22: The AAAI Conference on Artificial Intelligence.

Paper Code

Space-Time Memory Network for Sounding Object Localization in Videos

Sizhe Li‡, Yapeng Tian‡, and Chenliang Xu

BMVC'21: The British Machine Vision Conference.

Paper Code Project

Video Matting via Consistency-Regularized Graph Neural Networks

Tiantian Wang, Sifei Liu, Yapeng Tian, Kai Li, and Ming-Hsuan Yang

ICCV'21: IEEE/CVF International Conference on Computer Vision.

Paper Code

Can audio-visual integration strengthen robustness under multimodal attacks?

Yapeng Tian and Chenliang Xu

CVPR'21: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation

Yapeng Tian, Di Hu, and Chenliang Xu

CVPR'21: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing

Yapeng Tian, Dingzeyu Li, and Chenliang Xu

ECCV'20 Spotlight: European Conference on Computer Vision.

Paper Code Data

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiaoyu Xiang‡, Yapeng Tian‡, Yulun Zhang, Yun Fu, Jan Allebach, and Chenliang Xu

CVPR'20: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Video Code

TDAN: Temporally Deformable Alignment Network for Video Super-Resolution

Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu

CVPR'20: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

This is the first work that uses deformable alignment to address video restoration.

Paper Video Code

Deep Audio Prior

Yapeng Tian, Chenliang Xu, and Dingzeyu Li

CVPRW'20: CVPR Workshops.

Paper Code Project

Residual Dense Network for Image Super-Resolution

Yulun Zhang, Yapeng Tian, Yu Kong , Bineng Zhong, Yun Fu

TPAMI'20: IEEE Transactions on Pattern Analysis and Machine Intelligence.

Paper Code

CFSNet: Toward a Controllable Feature Space for Image Restoration

Wei Wang‡, Ruiming Guo‡, Yapeng Tian, and Wenming Yang

ICCV'19: IEEE/CVF International Conference on Computer Vision.

Paper Code

Interpretable and Controllable Audio-Visual Video Captioning

Yapeng Tian, Chenxiao Guan, Goodman Justin, Marc Moore, and Chenliang Xu

CVPRW'19: CVPR Workshops.

Multisensory interpretability in terms of the audio-visual video captioning task.

Paper

LCSCNet: Linear Compressing Based Skip-Connecting Network for ISR

Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang, Jing-Hao Xue, Qingmin Liao

TIP'19: IEEE Trans. Image Processing.

Paper

Deep Learning for Single Image Super-Resolution: A Brief Review

Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang, JingHao Xue, Qingmin Liao

TMM'19: IEEE Trans. Multimedia.

Paper

Audio-Visual Event Localization in Unconstrained Videos

Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, Chenliang Xu

ECCV'18: European Conference on Computer Vision.

Paper Video Code Data Project

Residual Dense Network for Image Super-Resolution

Yulun Zhang, Yapeng Tian, Yu Kong , Bineng Zhong, Yun Fu

CVPR'18 Spotlight: IEEE/CVF Conf. on Computer Vision and Pattern Recognition.

Paper Code

NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results

Timofte et al.

CVPRW'17: CVPR Workshops.

Paper

Consistent Coding Scheme for Single-Image Super-Resolution

Wenming Yang, Yapeng Tian, Fei Zhou, Qingmin Liao, Hai Chen, Chenglin Zheng

TMM'16: EEE Trans. Multimedia. (First student author)

Paper

Anchored Neighborhood Regression based SISR from Self-examples

Yapeng Tian, Fei Zhou, Wenming Yang, Xuesen Shang, Qingmin Liao

ICIP'16: IEEE International Conference on Image Processing.

Paper Code

SISR Using Clustering-Based Global Regression and Propagation Filtering

Wenming Yang, Yapeng Tian, Fei Zhou, ..., Qingmin Liao

ACPR'15 Oral: Asian Conference on Pattern Recognition. (First student author)

Paper

Towards Online Multi-Modal Social Interaction Understanding

Xinpeng Li, Shijian Deng, Bolin Lai, Weiguo Pian, James M. Rehg, Yapeng Tian

Preprint'25.

Paper

Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?

Jia Li, Wenjie Zhao, Ziru Huang, Yunhui Guo, Yapeng Tian

Preprint'25.

Paper

Modality-Inconsistent Continual Learning of Multimodal Large Language Models

Weiguo Pian, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

Preprint'24.

Paper

Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach

Shijian Deng, Wentian Zhao, Yu-Jhe Li, Kun Wan, Daniel Miranda, Ajinkya Kale, Yapeng Tian

Preprint'24.

Paper

AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation

Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, Yapeng Tian

Preprint'24.

Paper

Text-to-Audio Generation Synchronized with Videos

Shentong Mo, Jing Shi, Yapeng Tian

Preprint'24.

Paper

Robust Active Speaker Detection in Noisy Environments

Siva Sai Nagender Vasireddy, Chenxu Zhang, Xiaohu Guo, Yapeng Tian

Preprint'24.

Paper

SignDiff: Learning Diffusion Models for American Sign Language Production

Sen Fang, Chunyu Sui, Xuedong Zhang, Yapeng Tian

Preprint'23.

Paper

Learning in Audio-visual Context: A Review, Analysis, and New Perspective

Yake Wei, Di Hu, Yapeng Tian, Xuelong Li

Preprint'22.

Paper Project

Towards Online Multi-Modal Social Interaction Understanding

Xinpeng Li, Shijian Deng, Bolin Lai, Weiguo Pian, James M. Rehg, Yapeng Tian

Preprint'25.

Paper

Do Audio-Visual Segmentation Models Truly Segment Sounding Objects?

Jia Li, Wenjie Zhao, Ziru Huang, Yunhui Guo, Yapeng Tian

Preprint'25.

Paper

Modality-Inconsistent Continual Learning of Multimodal Large Language Models

Weiguo Pian, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

Preprint'24.

Paper

AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation

Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, Yapeng Tian

Preprint'24.

Paper

Text-to-Audio Generation Synchronized with Videos

Shentong Mo, Jing Shi, Yapeng Tian

Preprint'24.

Paper

Robust Active Speaker Detection in Noisy Environments

Siva Sai Nagender Vasireddy, Chenxu Zhang, Xiaohu Guo, Yapeng Tian

Preprint'24.

Paper

SignDiff: Learning Diffusion Models for American Sign Language Production

Sen Fang, Chunyu Sui, Xuedong Zhang, Yapeng Tian

Preprint'23.

Paper

Learning in Audio-visual Context: A Review, Analysis, and New Perspective

Yake Wei, Di Hu, Yapeng Tian, Xuelong Li

Preprint'22.

Paper Project

VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation

Saksham Singh Kushwaha, Yapeng Tian

CVPR'25: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Project

Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models

Saksham Singh Kushwaha, Jianbo Ma, Mark R. P. Thomas, Yapeng Tian, and Avery Bruni

ICASSP'25: IEEE International Conference on Acoustics, Speech, and Signal Processing.

Paper

Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters

Steven Hogue, Chenxu Zhang, Yapeng Tian, Xiaohu Guo

WACV'25: IEEE/CVF Winter Conference on Applications of Computer Vision.

Paper

Audio-Visual Dataset Distillation

Saksham Singh Kushwaha, Siva Sai Nagender Vasireddy, Kai Wang, Yapeng Tian

TMLR'24: Transactions on Machine Learning Research

Paper

Continual Audio-Visual Sound Separation

Weiguo Pian, Yiyang Nan, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

NeurIPS'24: The Annual Conference on Neural Information Processing Systems

Paper

Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition

Shijian Deng, Erin Kosloski, Siddhi Patel, Zeke A Barnett, Yiyang Nan, Alexander M Kaplan, Sisira Aarukapalli, William Doan, Matthew Wang, Harsh Singh, Rollins Pamela, Yapeng Tian

TMM'24: IEEE Transactions on Multimedia.

Paper Project

emoji_events DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models
(Best Paper Honorable Mention)

Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, Chenliang Xu

ACCV'24 Oral: Asian Conference on Computer Vision.

Paper

Language-Guided Joint Audio-Visual Editing Via One-Shot Adaptation

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

ACCV'24: Asian Conference on Computer Vision.

Paper

SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering

Tianyu Yang, Yiyang Nan, Lisen Dai, Zhenwen Liang, Yapeng Tian, Xiangliang Zhang

EMNLP'24: Empirical Methods in Natural Language Processing (Findings)

Paper

Towards Long Form Audio-visual Video Understanding

Wenxuan Hou, Guangyao Li, Yapeng Tian, Di Hu

TOMM'24: ACM Trans. on Multimedia Computing, Communications and App.

Paper Project

MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on Videos

Zheng Ning, Zheng Zhang, Jerrick Ban, Kaiwen Jiang, Ruohong Gan, Yapeng Tian, Toby Jia-Jun Li

C&C'24: ACM Conference on Creativity & Cognition.

Paper

AV-Mamba: Cross-Modality Selective State Space Models for Audio-Visual Question Answering

Ziru Huang, Jia Li, Wenjie Zhao, Yunhui Guo, Yapeng Tian

CVPRW'24: CVPR Signt and Sound Workshop

Paper

Learning Continual Audio-Visual Sound Separation Models

Weiguo Pian, Yiyang Nan, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian

CVPRW'24: CVPR Signt and Sound Workshop

Paper

Audio-Visual Autism Behavior Recognition with MMLMs

Shijian Deng, Erin Kosloski, Siddhi Patel, Zeke A Barnett, Yiyang Nan, Alexander M Kaplan, Sisira Aarukapalli, William Doan, Matthew Wang, Harsh Singh, Rollins Pamela, Yapeng Tian

CVPRW'24: CVPR Signt and Sound Workshop

Paper

Dataset distillation for audio-visual datasets

Saksham Singh Kushwaha, Siva Sai Nagender Vasireddy, Kai Wang, Yapeng Tian

CVPRW'24: CVPR Signt and Sound Workshop

Paper

DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures

Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, Xiaohu Guo

CVPRW'24: CVPR HuMoGen Workshop

Paper

Towards Efficient Audio-Visual Learners via Empowering Pre-trained Vision Transformers with Cross-Modal Adaptation

Kai Wang, Yapeng Tian, Dimitrios Hatzinakos

CVPRW'24: CVPR Multimodal Foundation Models Workshop

Paper Code

MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers

Tanvir Mahmud, Shentong Mo, Yapeng Tian, Diana Marculescu

CVPRW'24: CVPR Efficient Deep Learning for Computer Vision Workshop

Paper

T-VSL: Text-Guided Visual Sound Source Localization in Mixtures

Tanvir Mahmud, Yapeng Tian, Diana Marculescu

CVPR'24: IEEE/CVF Conference on Computer Vision and Pattern Recognition

Paper Code

SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision Viewers

Zheng Ning, Brianna Wimer, Kaiwen Jiang, Keyi Chen, Jerrick Ban, Yapeng Tian, Yuhang Zhao, Toby Li

CHI'24: The ACM Conference on Human Factors in Computing Systems.

Paper

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Yuxin Ye, Wenming Yang, Yapeng Tian

WACV'24: Winter Conference on Applications of Computer Vision.

Paper Project

Disentangled counterfactual learning for physical audiovisual commonsense reasoning

Changsheng Lv, Shuai Zhang, Yapeng Tian, Mengshi Qi, Huadong Ma

NeurIPS'23: The Annual Conference on Neural Information Processing Systems.

Paper Code

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

NeurIPS'23: The Annual Conference on Neural Information Processing Systems.

Paper Project

PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data

Zheng Zhang‡, Zheng Ning‡, Chenliang Xu Yapeng Tian, Toby Jia-Jun Li

UIST'23: ACM Symposium on User Interface Software and Technology.

Paper

Towards Robust Active Speaker Detection

Siva Sai Nagender Vasireddy, Chenxu Zhang, Xiaohu Guo, Yapeng Tian

ICCVW'23: ICCV AV4D Workshop .

Paper

Position-Aware Audio-Visual Separation for Spatial Audio

Yuxin Ye, Wenming Yang, Yapeng Tian

ICCVW'23: ICCV AV4D Workshop .

Paper

Towards Better Egocentric Action Understanding in a Multi-Input Multi-Output View

Wenxuan Hou, Ruoxuan Feng, Yixin Xu, Yapeng Tian, Di Hu

ICCVW'23: ICCV AV4D Workshop .

Paper

Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

ICCVW'23: ICCV AV4D Workshop .

Paper

Separating Invisible Sounds Toward Universal Audio-Visual Scene-Aware Sound Separation

Yiyang Su, Ali Vosoughi, Shijian Deng, Yapeng Tian, Chenliang Xu

ICCVW'23: ICCV AV4D Workshop .

Paper

Audio-Visual Class-Incremental Learning

Weiguo Pian‡, Shentong Mo‡, Yunhui Guo, Yapeng Tian

ICCV'23: IEEE/CVF International Conference on Computer Vision.

Paper Code

Class-Incremental Grouping Network for Continual Audio-Visual Learning

Shentong Mo‡, Weiguo Pian‡, Yapeng Tian

ICCV'23: IEEE/CVF International Conference on Computer Vision.

Paper Code

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Shentong Mo, Yapeng Tian

CVPRW'23: CVPR Sight and Sound Workshop.

Paper

DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment

Shentong Mo, Jing Shi, Yapeng Tian

CVPRW'23: CVPR Sight and Sound Workshop.

Paper

AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

CVPRW'23: CVPR Sight and Sound Workshop.

Paper Project

Audio-Visual Grouping Network for Sound Localization from Mixtures

Shentong Mo, Yapeng Tian

CVPR'23: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Egocentric Audio-Visual Object Localization

Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu

CVPR'23: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Towards Unified, Explainable, and Robust Multisensory Perception

Yapeng Tian

AAAI'23: AAAI Conference on Artificial Intelligence. (NFH program)

Paper

Multi-modal Grouping Network for Weakly-Supervised Audio-Visual Video Parsing

Shentong Mo, Yapeng Tian

NeurIPS'22: The Annual Conference on Neural Information Processing Systems.

Paper Code

Audio-Visual Scene Understanding Towards Unified, Explainable, and Robust Multisensory Perception

Yapeng Tian

PhD Thesis

Paper

Audio-Visual Object Localization in Egocentric Videos

Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu

CVPRW'22: CVPR Workshops

Egocentric audio-visual learning.

Paper

Learning to Answer Questions in Dynamic Audio-Visual Scenarios

Guangyao Li‡, Yake Wei‡, Yapeng Tian‡, Chenliang Xu, Ji-Rong Wen, and Di Hu

CVPR'22 Oral: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Video Code Project

Space-Time Memory Network for Sounding Object Localization in Videos

Sizhe Li‡, Yapeng Tian‡, and Chenliang Xu

BMVC'21: The British Machine Vision Conference.

Paper Code Project

Can audio-visual integration strengthen robustness under multimodal attacks?

Yapeng Tian and Chenliang Xu

CVPR'21: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation

Yapeng Tian, Di Hu, and Chenliang Xu

CVPR'21: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing

Yapeng Tian, Dingzeyu Li, and Chenliang Xu

ECCV'20 Spotlight: European Conference on Computer Vision.

Paper Code Data

Deep Audio Prior

Yapeng Tian, Chenliang Xu, and Dingzeyu Li

CVPRW'20: CVPR Workshops.

Paper Code Project

Interpretable and Controllable Audio-Visual Video Captioning

Yapeng Tian, Chenxiao Guan, Goodman Justin, Marc Moore, and Chenliang Xu

CVPRW'19: CVPR Workshops.

Multisensory interpretability in terms of the audio-visual video captioning task.

Paper

Audio-Visual Event Localization in Unconstrained Videos

Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, Chenliang Xu

ECCV'18: European Conference on Computer Vision.

Paper Video Code Data Project

EgoVSR: Towards High-Quality Egocentric Video Super-Resolution

Yichen Chi, Junhao Gu, Jiamiao Zhang, Wenming Yang, Yapeng Tian

TCSVT'24: IEEE Transactions on Circuits and Systems for Video Technology.

Paper

STADNet: Spatial-Temporal Attention-Guided Dual-Path Network for cardiac cine MRI super-resolution

Jun Lyu, Shuo Wang, Yapeng Tian, Jing Zou, Shunjie Dong, Chengyan Wang, Angelica I Aviles-Rivero, Jing Qin

MIA'24: Medical Image Analysis

Paper

Structured Sparsity Learning for Efficient Video Super-Resolution

Bin Xia, Jingwen He, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Luc Van Gool

CVPR'23: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Stdan: deformable attention network for space-time video super-resolution

Hai Wang, Xiaoyu Xiang, Yapeng Tian, Wenming Yang, Qingmin Liao

TNNLS'23: IEEE Transactions on Neural Networks and Learning Systems.

Paper Code

Learning Spatio-Temporal Downsampling for Effective Video Upscaling

Xiaoyu Xiang, Yapeng Tian, Vijay Rengarajan, Lucas Young, Bo Zhu, Rakesh Ranjan

ECCV'22: European Conference on Computer Vision.

Paper

Video Matting via Consistency-Regularized Graph Neural Networks

Tiantian Wang, Sifei Liu, Yapeng Tian, Kai Li, and Ming-Hsuan Yang

ICCV'21: IEEE/CVF International Conference on Computer Vision.

Paper Code

Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution

Xiaoyu Xiang‡, Yapeng Tian‡, Yulun Zhang, Yun Fu, Jan Allebach, and Chenliang Xu

CVPR'20: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Video Code

TDAN: Temporally Deformable Alignment Network for Video Super-Resolution

Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu

CVPR'20: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

This is the first work that uses deformable alignment to address video restoration.

Paper Video Code

DiffIR: Efficient Diffusion Model for Image Restoration

Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, Radu Timotfe, Luc Van Gool

TPAMI'25: IEEE Transactions on Pattern Analysis and Machine Intelligence.

Paper Code

DiffIR: Efficient Diffusion Model for Image Restoration

Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, Luc Van Gool

ICCV'23: IEEE/CVF International Conference on Computer Vision.

Paper Code

Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI

Jiamiao Zhang, Yichen Chi, Jun Lyu, Wenming Yang, Yapeng Tian

MICCAI'23: Medical Image Computing and Computer-Assisted Intervention.

Paper Code

Meta-Learning based Degradation Representation for Blind Super-Resolution

Bin Xia, Yapeng Tian, Yulun Zhang, Yucheng Hang, Wenming Yang, Qingmin Liao

TIP'23: IEEE Transactions on Image Processing.

Paper Code

Knowledge Distillation based Degradation Estimation for Blind Super-Resolution

Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, Luc Van Gool

ICLR'23: International Conference on Learning Representations.

Paper Code

Basic Binary Convolution Unit for Binarized Image Restoration Network

Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, Luc Van Gool

ICLR'23: International Conference on Learning Representations.

Paper Code

GDSSR: Toward Real-World Ultra-High-Resolution Image Super-Resolution

Yichen Chi, Wenming Yang, Yapeng Tian

SPL'23: IEEE Signal Processing Letters.

Paper

DuDoCAF: Dual-Domain Cross-Attention Fusion with Recurrent Transformer for Fast Multi-contrast MR Imaging

Jun Lyu, Bin Sui, Chengyan Wang, Yapeng Tian, Qi Dou, and Jing Qin

MICCAI'22: Medical Image Computing and Computer Assisted Intervention.

Paper

Transformer-empowered Multi-contrast MRI Super-Resolution

Guangyuan Li, Jun Lv, Yapeng Tian, Qi Dou, Chengyan Wang, Chenliang Xu, Jing Qin

CVPR'22: IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Paper Code

Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-based Super-Resolution

Bin Xia, Yapeng Tian, Yucheng Hang, Wenming Yang, Qingmin Liao, Jie Zhou

AAAI'22: The AAAI Conference on Artificial Intelligence.

Paper Code

Efficient Non-Local Contrastive Attention for Image Super-Resolution

Bin Xia‡, Yucheng Hang‡, Yapeng Tian, Wenming Yang, Qingmin Liao, Jie Zhou

AAAI'22: The AAAI Conference on Artificial Intelligence.

Paper Code

Residual Dense Network for Image Super-Resolution

Yulun Zhang, Yapeng Tian, Yu Kong , Bineng Zhong, Yun Fu

TPAMI'20: IEEE Transactions on Pattern Analysis and Machine Intelligence.

Paper Code

CFSNet: Toward a Controllable Feature Space for Image Restoration

Wei Wang‡, Ruiming Guo‡, Yapeng Tian, and Wenming Yang

ICCV'19: IEEE/CVF International Conference on Computer Vision.

Paper Code

LCSCNet: Linear Compressing Based Skip-Connecting Network for ISR

Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang, Jing-Hao Xue, Qingmin Liao

TIP'19: IEEE Trans. Image Processing.

Paper

Deep Learning for Single Image Super-Resolution: A Brief Review

Wenming Yang, Xuechen Zhang, Yapeng Tian, Wei Wang, JingHao Xue, Qingmin Liao

TMM'19: IEEE Trans. Multimedia.

Paper

Residual Dense Network for Image Super-Resolution

Yulun Zhang, Yapeng Tian, Yu Kong , Bineng Zhong, Yun Fu

CVPR'18 Spotlight: IEEE/CVF Conf. on Computer Vision and Pattern Recognition.

Paper Code

NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results

Timofte et al.

CVPRW'17: CVPR Workshops.

Paper

Consistent Coding Scheme for Single-Image Super-Resolution

Wenming Yang, Yapeng Tian, Fei Zhou, Qingmin Liao, Hai Chen, Chenglin Zheng

TMM'16: EEE Trans. Multimedia. (First student author)

Paper

Anchored Neighborhood Regression based SISR from Self-examples

Yapeng Tian, Fei Zhou, Wenming Yang, Xuesen Shang, Qingmin Liao

ICIP'16: IEEE International Conference on Image Processing.

Paper Code

SISR Using Clustering-Based Global Regression and Propagation Filtering

Wenming Yang, Yapeng Tian, Fei Zhou, ..., Qingmin Liao

ACPR'15 Oral: Asian Conference on Pattern Recognition. (First student author)

Paper

Teaching

Service

Organizer:

Senior Program Committee or Area Chair:

  • NeurIPS: Conference on Neural Information Processing Systems, 2025
  • CVPR: IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025
  • ICLR: International Conference on Learning Representations, 2025
  • AAAI: AAAI Conference on Artificial Intelligence, 2023, 2024, 2025

Session Chair:

  • AAAI 2023 (Multimodal Learning, Low-Level & Physics-based Vision)

Conference Program Committee/Reviewer:

  • CVPR: IEEE/CVF Conference on Computer Vision and Pattern Recognition
  • ICCV: IEEE/CVF International Conference on Computer Vision
  • ECCV: European Conference on Computer Vision
  • NeurIPS: Conference on Neural Information Processing Systems
  • ICLR: International Conference on Learning Representations
  • AAAI: AAAI Conference on Artificial Intelligence
  • ICML: International Conference on Machine Learning
  • WACV: Winter Conference on Applications of Computer Vision
  • ACCV: Asian Conference on Computer Vision
  • MICCAI: International Conference On Medical Image Computing & Computer Assisted Intervention

Journal Reviewer:

  • TPAMI: IEEE Transactions on Pattern Analysis and Machine Intelligence
  • TMLR: The Transactions on Machine Learning Research
  • TIP: IEEE Transactions on Image Processing
  • TNNLS: IEEE Transactions on Neural Networks and Learning Systems
  • TMM: IEEE Transactions on Multimedia
  • TCSVT: IEEE Transcations on Circuits and Systems for Video Technology
  • TASLP: IEEE/ACM Transactions on Audio, Speech and Language Processing
  • Scientific Reports–Nature
  • CGF: Computer Graphics Forum
  • CVIU: Computer Vision and Image Understanding
  • SPIC: Signal Processing: Image Communication
  • IEEE Access

Talks and Seminars:

  • Audio-visual Scene Perception and Generation
         AIM Seminar @ UNT AI Seminar, Jan. 2025
  • Enhancing Image Quality with Deep Learning-Based Super-Resolution: From Natural Scenes to Medical Imaging
         AIM Seminar @ UTSW, Oct. 2024
  • Learning Semantic-aware Grouping for Weakly-Supervised Audio-Visual Scene Understanding
         Sight and Sound Workshop @ CVPR, June 2023
  • Human-Multisensory AI Collaboration: Opportunities and Challenges
          AV4D Workshop @ ECCV, Oct. 2022
  • UTD CS Mixer, Oct. 2022
  • Audio-Visual Scene Understanding Towards Unified, Explainable, and Robust Multisensory Perception

    KTH Dive-Deep Seminar, Dec. 2021
         RIT PhD Colloquium Series, Oct. 2021

  • Audio-Visual Video Understanding, IIAI Seminar, Sep. 2021
  • The Future of Audio-Visual Research Panel Discussion, VALSE Webinar, Nov. 2021

Awards

ACCV Best Paper Honorable Mention Award, 2024
IEEE ISMAR IDEATExR workshop Best Paper Award, 2024
UIST Belonging & Inclusion Best Paper Award, 2024
Amazon Research Award, 2024
Undergraduate Research Apprenticeship Program (URAP) award, 2023 and 2024
Cisco Faculty Research Award, 2023
AAAI New Faculty Highlights, 2023
CVPR Doctoral Consortium, 2022
Top 10% of High-Scoring Reviewers for NeurIPS, 2020
Outstanding Graduate of Tsinghua University, 2017
Outstanding Master Thesis Award, Tsinghua University, 2017
National Scholarship, Tsinghua University, 2016

Vitæ

Full CV in PDF.

  • University of Texas at Dallas 2022 - now
    Assistant Professor
    Department of Computer Science
  • University of Rochester 2017 - 2022
    Ph.D. Student
    Department of Computer Science
  • Meta Sep. 2021 - Jan. 2022
    Research Intern
    Reality Labs
  • Adobe Summer 2021
    Research Intern
    Creative Intelligence Lab
  • Adobe Summer 2019
    Research Intern
    Creative Intelligence Lab
  • Tsinghua University 2014-2017
    M.E. Student
    Department of Electronic Engineering
  • Chinese Academy of Sciences Nov. 2016- May 2017
    Visiting Student
    Shenzhen Institutes of Advanced Technology
  • Xidian University 2009 - 2013
    B.E. Student
    School of Electronic Engineering

This website was built with jekyll based on a template from Martin Saveski.

Yapeng Tian
yapeng.tian@utdallas.edu