Sheng Shen

Ph.D. student in BAIR, EECS at the University of California, Berkeley.

Berkeley, California
Email: sheng.s@berkeley.edu
Google Scholar
Github
Twitter
Linkedin

At Berkeley, I am advised by Prof. Kurt Keutzer and Prof. Trevor Darrell. I also work closely with Prof. Dan Klein and Prof. Michael Mahoney. Prior to Berkeley, I received my bachelor degree in computer science from Peking University, advised by Prof. Xuanzhe Liu. I have received the Lotfi A. Zadeh Prize for my research work.
My research interests focus on compute-optimal (multimodal) language modeling, including efficient training/tuning methods, model compression techniques, and the integration of vision-language models.

Preprints

Aligning Large Multimodal Models with Factually Augmented RLHF

Zhiqing Sun^*, Sheng Shen^*, Shengcao Cao^*, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan^†, Liang-Yan Gui^†, Yu-Xiong Wang^†, Yiming Yang^†, Kurt Keutzer^†, Trevor Darrell^†

Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models

Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, Denny Zhou

Publications (selected)

K-LITE: Learning Transferable Visual Models with External Knowledge NeurIPS 2022

Sheng Shen^*, Chunyuan Li^*, Xiaowei Hu^*, Yujia Xie, Jianwei Yang, Pengchuan Zhang, Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, Kurt Keutzer, Trevor Darrell, Anna Rohrbach, Jianfeng Gao
[Code]

Staged Training for Transformer Language Models ICML 2022

Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy
[Code]

How Much Can CLIP Benefit Vision-and-Language Tasks? ICLR 2022

Sheng Shen^*, Liunian Harold Li^*, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer [Code]

Multitask prompted training enables zero-shot task generalization ICLR 2022

Hugging Face + Big Science
[Code]

Learned Token Pruning for Transformers KDD 2022

Sehoon Kim^*, Sheng Shen^*, David Thorsley^*, Amir Gholami^*, Joseph Hassoun, Kurt Keutzer
[Code]

Reservoir Transformers ACL 2021

Sheng Shen, Alexei Baevski, Ari S. Morcos, Kurt Keutzer, Michael Auli and Douwe Kiela

ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning AAAI 2021

Zhewei Yao^*, Amir Gholami^*, Sheng Shen, Kurt Keutzer and Michael Mahoney
[Code]

Noisy Self-Knowledge Distillation for Text Summarization NAACL 2021

Yang Liu, Sheng Shen and Mirella Lapata
[Code] [Video] [Poster]

PowerNorm: Rethinking Batch Normalization in Transformers ICML 2020

Sheng Shen^*, Zhewei Yao^*, Amir Gholami, Michael Mahoney and Kurt Keutzer
[Code] [Slides]

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers ICML 2020

Zhuohan Li^*, Eric Wallace^*, Sheng Shen^*, Kevin Lin^*, Kurt Keutzer, Dan Klein and Joseph E. Gonzalez.
[Slides] [Blog] [Misc]

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT AAAI 2020

Sheng Shen^*, Zhen Dong^*, Jiayu Ye^*, Linjian Ma, Zhewei Yao, Amir Gholami, Michael Mahoney and Kurt Keutzer
[Code]

Pragmatically Informative Text Generation NAACL 2019 short

Sheng Shen, Daniel Fried, Jacob Andreas and Dan Klein
[Code] [Slides]

Ermes: Emoji-Powered Representation Learning for Cross-Lingual Sentiment Classification WWW 2019

Zhenpeng Chen^*, Sheng Shen^*, Ziniu Hu, Xuan Lu, Xuanzhe Liu and Qiaozhu Mei
[Code] [Slides] [Best Paper Award]

Experience

Google, Student Researcher
Advised by Le Hou and Denny Zhou, Mar. 2023 - Aug. 2023

Microsft, Research Intern
Advised by Zhewei Yao and Chunyuan Li, Feb. 2022 - Aug. 2022

Allen Institute for Artificial Intelligence, Research Intern
Advised by Iz Beltagy, Matthew Peters and Jesse Dodge, May. 2021 - Aug. 2021

Facebook AI Research, Research Intern
Advised by Douwe Kiela and Michael Auli, May. 2020 - Dec. 2020

Berkeley AI Research, Junior Specialist II
Advised by Prof. Kurt Keutzer, Prof. Dan Klein and Prof. Michael Mahoney, Jun. 2019 - May. 2020

Tencent AI Lab, Research Intern
Advised by Yaliang Li and Wei Fan, Apr. 2018 - Sept. 2018

University of Illinois at Urbana-Champaign, Research Intern
Advised by Prof. Aditya Parameswaran, Jun. 2017 - Sept. 2017

Teaching

CS267 Applications of Parallel Computers, Spring 2023
CS282 Deep Neural Networks, Fall 2022