Ambient intelligence for mobile, wearable, IoT and Robotics systems

Ke Sun

Assistant Professor, Computer Science and Engineering, EECS Department, University of Michigan, Ann Arbor

kesuniot@umich.edu

Samsonsjarkal

Google Scholar

Resume

Ambient Intelligence (AmI) Lab

Personal photo

Biography

Ke Sun is an Assistant Professor in the CSE Division of the EECS Department at the University of Michigan, Ann Arbor, where he leads the Ambient Intelligence (AmI) Lab.

Committed to the vision of "ambient intelligence," his research focuses on computational sensing systems that are intelligent, deployable, and human-centric across mobile, wearable, IoT, and robotic platforms. His work is driven by three core research questions:

  • How can off-the-shelf sensors be repurposed to achieve extended sensing capabilities?
  • How can we address the system challenges of emerging AI-native sensing systems?
  • How can sensing systems be designed to better support everyday life?

His work has received the ACM IMWUT (UbiComp) Distinguished Paper Award and the ACM SenSys Best Poster Runner-up Award, and he is also a recipient of the Google Ph.D. Fellowship. Its real-world impact is reflected in successful deployments on widely used IoT platforms such as Amazon Echo and Google Home.

His research is organized around two broad directions:

  • New Multi-modal Computational Sensing Technologies for Diverse Applications:

    • HCI and Ubiquitous Computing: UWB-Prac (SenSys'26), UltraPoser (UIST'25), SCALAR (TMC'23), UltraSE (MobiCom'21), ExGSense (IPSN'21), DSW (INFOCOM'20), VSKIN (MobiCom'18), Dolphin (MobiSys'18), and LLAP (MobiCom'16).

    Health Sensing: EarCardio (MobiCom'26), EgoADL (UbiComp'24), LoEar (UbiComp'22), and Unlock With Your Heart (UbiComp'18).

    Environmental Sensing: MoiréLens (SenSys'26) and VECTOR (UbiComp'22).

    Robotic Sensing: RFCanvas (SenSys'24) and milliEgo (SenSys'20).

  • AI-Native Sensing Systems Challenges for Mobile, Wearable, Robotic, and IoT Platforms:

    • System Optimization: improving efficiency, latency, computational overhead, storage, and communication for practical sensing systems: KDC (NSDI'26), RFCanvas (SenSys'24), and SCALAR (TMC'23).

    Security and Privacy: building secure and trustworthy sensing platforms through both attack analysis and defense design: MagicPatch (MobiCom'26), EveGuard (S&P'25), LCCT (AAAI'25), Magmaw (NDSS'25), StealthyIMU (NDSS'23), MicShield (SenSys'20), and SpiderMon (INFOCOM'20).

News

Selected Publication (Full List)

A note about authors list: Underlined are students that I supervise. '*' marks co-primary authors.

MoiréLens: Bringing Schlieren Imaging into Real-World Environments Using Moiré Patterns
Linzhen Zhu*, Runqiu Wang*, Yi Rong, Ke Sun
ACM SenSys 2026, May 2026. (Acceptance ratio: 18.9%)

MoiréLens demonstrates that everyday cameras can be used for visualizing air-flow and refractive-index dynamics. It converts subtle refractive-index variations into measurable Moiré phase shifts formed between the background and the camera’s color-filter array.
moirelens

Unlocking Practical Cardiac Monitoring Capabilities on True Wireless Stereo Earbuds (To Appear)
Yongjian Fu, Ke Sun, Xinyu Zhang, Ruyao Wang, Xinyi Li, Hao Pan, Yaoxue Zhang, Ju Ren
ACM MobiCom 2026, May 2026. (Acceptance ratio: 11.0%)

EarCardio repurposes the IMU sensors in TWS earbuds for cardiac monitoring by capturing in-ear ballistocardiogram (BCG) signals. Our experiments on 110 subjects verify that EarCardio can accurately reconstruct cardiac signals while showing resilience to motion artifacts, missing data, and low sampling rates.
earcardio


Attacking mmWave Imaging Using Metasurfaces Based on Intensity and Time-of-Flight Manipulation (To Appear)
Shuning Wang, Yongjian Fu, Hao Pan, Xinyi Li, Linghui Zhong, Yaxiong Xie, Ke Sun, Guangtao Xue, Yaoxue Zhang, Ju Ren
ACM MobiCom 2026, May 2026. (Acceptance ratio: 11.0%)

MagicPatch proposes an efficient attack on millimeterwave (mmWave) security screening systems by designing a novel reflective metasurface patch that can be easily attached to clothing.
magicPatch


SoilNutri: A Passive Metasurface-based, Low-cost System for Soil Moisture and Nitrogen Monitoring
Juexing Wang, Binbin Xie, Yimeng Liu, Minhao Cui, Xiao Zhang, Guangjing Wang, Ke Sun, Zhichao Cao, Huacheng Zeng, Qingxu Jin, et al.
ACM IMWUT (UbiComp) 2026, Mar 2026.

SoilNutri is a passive metasurface-based and low-cost system for soil moisture and nitrogen monitoring.
SoilNutri


From Bits to Tokens: Knowledge-Driven Generative Communication of Multimodal Data (To Appear)
Xingyu Chen, Zihao Feng, Wuqiong Zhao, Jianrong Ding, Ke Sun, Xinyu Zhang
USENIX NSDI 2026, May 2026. (Acceptance ratio: 22.1%)

We introduce Knowledge-Driven Communication (KDC), a framework that transmits semantic knowledge rather than raw bits by leveraging pretrained knowledge bases.
kdc


UWB-based Localization of Smartphones Inside a Vehicle to Prevent Distracted Driving (To Appear)
Kailai Cui, Ke Sun, Kang G. Shin
ACM SenSys 2026, May 2026. (Acceptance ratio: 19.0%)

UWB-Prac is a real-time seat-level smartphone localization system that operates using only the existing single UWB anchor infrastructure commonly found in modern vehicles. It localizes smartphone in real-time during driving and resolves spatial ambiguity in CIR features by: (1) optimizing UWB anchor placement during vehicle design, and (2) applying a directional RF shield in retrofit scenarios.
uwbprac


UltraPoser: Pushing the Limits of IMU-based Full-Body Pose Estimation with Ultrasound Sensing on Consumer Wearables
Yadong Li*, Shuning Wang*, Yongjian Fu, Justin Chen, Xingyu Chen, Ju Ren, Xinyu Zhang, Akshay Gadre, Ke Sun
ACM UIST 2025, Oct 2025. (Acceptance ratio: 22.2%)

UltraPoser leverages built-in microphones and speakers on commodity wearables, such as smartphones and smartwatches, to transmit and receive inaudible ultrasound signals, to support IMU-based full-body pose estimation with high accuracy.
ultraposer


EveGuard: Defeating Vibration-based Side-Channel Eavesdropping with Audio Adversarial Perturbations
Jung-woo Chang*, Ke Sun*, David Xia, Xinyu Zhang, Farinaz Koushanfar
IEEE S&P (Oakland) 2025, May 2025. (Acceptance ratio: 14.9%)

Vibrometry-based side channels pose a significant privacy risk, exploiting sensors like mmWave radars, light sensors, and accelerometers to detect vibrations from sound sources or proximate objects, enabling speech eavesdropping. This paper presents EveGuard, a software-driven defense framework that creates adversarial audio, protecting voice privacy from these side channels without compromising human perception.
eveguard


Security Attacks on LLM-based Code Completion Tools
Cheng Wen, Ke Sun, Xinyu Zhang, Wei Wang
AAAI 2025 (Oral), Feb 2025. (Top 5%)

This paper exploits these distinct security vulnerabilities of LLM-based Code Completion Tools to develop targeted attack methodologies on two critical security risks: jailbreaking and training data extraction attacks.
lcct


Magmaw: Modality-Agnostic Adversarial Attacks on Machine Learning-Based Wireless Communication Systems
Jung-Woo Chang, Ke Sun, Nasimeh Heydaribeni, Seira Hidano, Xinyu Zhang, Farinaz Koushanfa
NDSS 2025, Feb 2025. (Acceptance ratio: 16.1%)

We propose Magmaw, a novel attack methodology targeting deep neural network (DNN)-based joint source-channel coding (JSCC) wireless communication systems. Magmaw is capable of generating universal adversarial perturbations for any multimodal signal transmitted over a wireless channel.
magmaw


RFCanvas: Modeling RF Channel by Fusing Visual Priors and Few-shot RF Measurements
Xingyu Chen, Zihao Feng, Ke Sun, Kun Qian, Xinyu Zhang
SenSys 2024, Nov 2024.

We propose RFCanvas, which fuses visual priors and RF measurements to achieve high accuracy for realistic scenes and be responsive to environmental changes.
rfcanvas


Multimodal Daily-life Logging in Free-living Environments Using Non-Visual Egocentric Sensors on a Smartphone
Ke Sun, Chunyu Xia, Xinyu Zhang, Hao Chen, Charlie Zhang
ACM IMWUT (UbiComp) 2024, March 2024.

We propose EgoADL, the first egocentric ADL sensing system that uses an in-pocket smartphone as a multi-modal sensor hub to capture body motion, interaction with the physical environment and daily objects using non-visual sensors (audio, wireless sensing, and motion sensors).
egoadl


StealthyIMU: Extracting Permission-protected Private Information from Smartphone Voice Assistant using Zero-Permission Sensors
Ke Sun, Chunyu Xia, Songlin Xu, Xinyu Zhang
NDSS 2023, Feb 2023. (Acceptance ratio: 58/398=14.6%)

We propose StealthyIMU, a new threat that uses side channel to steal permission-protected private information from the voice-user interface.
stealthyimu


SCALAR: Self-Calibrated Acoustic Ranging for Distributed Mobile Devices
Lei Wang, Haoran Wan, Ting Zhao, Ke Sun, Shuyu Shi, Haipeng Dai, Guihai Chen, Haodong Liu, Wei Wang
IEEE TMC 2023

We propose SCALAR, a self-calibrated acoustic ranging system that achieves sub-millimeter ranging accuracy on distributed commercial devices.
scalar


VECTOR: Velocity Based Temperature-field Monitoring with Distributed Acoustic Devices
Haoran Wan, Lei Wang, Ting Zhao, Ke Sun, Shuyu Shi, Haipeng Dai, Guihai Chen, Haodong Liu, Wei Wang
ACM IMWUT (UbiComp) 2022, Sep 2022.

We introduce VECTOR, a temperature-field monitoring system that achieves high temperature sensing accuracy and fast response time using commercial sound playing/recording devices.
vector


LoEar: Push the Range Limit of Acoustic Sensing for Vital Sign Monitoring
Lei Wang, Wei Li, Ke Sun, Fusang Zhang, Tao Gu, Chenren Xu, Daqing Zhang
ACM IMWUT (UbiComp) 2022, Sep 2022.

We propose LoEar, which is a novel acoustic sensing system using only a single microphone and speaker to detect vital signs (respiration and heartbeat) with a significantly increased sensing range (> 6 meter).
loear


UltraSE: Single-Channel Speech Enhancement Using Ultrasound
Ke Sun, Xinyu Zhang
ACM MobiCom 2021, Oct 2021. (Acceptance ratio: 60/299=20.1%)

We propose UltraSE, which uses ultrasound sensing as a complementary modality to separate the desired speaker’s voice from interferences and noise.
ultrase


ExGSense: Toward Facial Gesture Sensing and Reconstruction with a Sparse Near-Eye Sensor Array
Chen Chen, Ke Sun, Xinyu Zhang
ACM/IEEE IPSN 2021, May 2021. (Acceptance ratio: 26/105=24.8%)

ExGSense propose to sense whole facial gestures by using a sparse set of biopotential transducers. We propose a dual-branch multiview representation learning model, which can explicitly exploit the sensor diversities across time-frequency-spatial domains.
exgsense


"Alexa, Stop Spying on Me": Speech Privacy Protection Against Voice Assistants
Ke Sun, Chen Chen, Xinyu Zhang
ACM SenSys 2020, Nov 2020. (Acceptance ratio: 43/213=20.2%)

MicShield is a companion device with voice assistant to enable the selective jamming mechanism which, for the first time, prevents VAs from recording private speech without affecting the voice assistants’ normal functionalities.
micshield


milliEgo: Single-chip mmWave Radar Aided Egomotion Estimation via Deep Sensor Fusion
Chris Xiaoxuan Lu, Muhamad Risqi U. Saputra, Peijun Zhao, Yasin Almalioglu, Pedro P. B. de Gusmao, Changhao Chen, Ke Sun, Niki Trigoni, Andrew Markham
ACM SenSys 2020, Nov 2020. (Acceptance ratio: 43/213=20.2%)

milliEgo is a novel deep-learning approach that demonstrates the feasibility of mmWave odometry and provides a fusion framework to develop a robust mmWaveInertial Odometry.
milliego


Dynamic Speed Warping: Similarity-Based One-shot Learning for Device-free Gesture Signals
Xun Wang, Ke Sun, Ting Zhao, Wei Wang, Qing Gu
IEEE INFOCOM 2020, April 2020. (Acceptance ratio: 267/1397=19.1%)

We propose a Dynamic Speed Warping (DSW) algorithm to enable one-shot learning for device-free gesture signals performed by different users. DSW provides a robust similarity measure between gesture samples and can work for both one-shot learning in supervised gesture recognition and unsupervised learning tasks.
dsw


SpiderMon: Towards Using Cell Towers as Illuminating Sources for Keystroke Monitoring
Kang Ling, Yuntang Liu, Ke Sun, Wei Wang, Lei Xie, Qing Gu
IEEE INFOCOM 2020, April 2020. (Acceptance ratio: 267/1397=19.1%)

SpiderMon is a system that performs longrange keystroke monitoring using the signal transmitted by commercial 4G/5G cellular base stations. We show that SpiderMon can detect keystroke movements at a distance of 15 meters or even behind the wall.
SpiderMon


VSkin: Sensing Touch Gestures on Surfaces of Mobile Devices Using Acoustic Signals
Ke Sun, Ting Zhao, Wei Wang, Lei Xie
ACM MobiCom 2018, October 2018 (Acceptance ratio: 42/187=22.5%).

VSkin is a system that supports fine-grained touch gesture sensing on the back of mobile devices based on acoustic signals. VSkin utilizes both the structure-borne sounds, i.e., sounds propagating through the structure of the device, and the air-borne sounds, i.e., sounds propagating through the air, to sense touch gestures.
vskin


Unlock With Your Heart: Heartbeat-based Authentication on Commercial Mobile Phones
Lei Wang, Kang Huang, Ke Sun, Wei Wang, Chen Tian, Lei Xie, Qing Gu
ACM IMWUT (UbiComp) 2018, October 2018.

''Unlock with your heart'' uses the built-in accelerometer to capture the heartbeat signals on commercial mobile phones. The user only needs to press the phone on his/her chest, and the system can identify the user within a few heartbeats.
heartbeat


Depth Aware Finger Tapping on Virtual Displays
Ke Sun, Wei Wang, Alex X. Liu, Haipeng Dai
ACM MobiSys 2018, June 2018. (Acceptance ratio: 37/138=26.8%)

Dolphin is a fine-grained depthaware tapping scheme that can provide high accuracy tapping detection. It uses light-weight ultrasound based sensing, along with one COTS mono-camera, to enable 3D tracking of fingers.
Dolphin


Device-free Gesture Tracking using Acoustic Signals
Wei Wang, Alex X. Liu, Ke Sun
ACM MobiCom 2016, October 2016. (Acceptance ratio: 31/226=13.7%)

LLAP is an ultrasound-based device-free gesture tracking scheme that can be deployed on existing mobile devices as software, without any hardware modification. We use speakers and microphones that already exist on most mobile devices to perform device-free tracking of a hand/finger.
MIT Technology Review: This App Lets You Control Your Phone Using Sonar
LLAP

Selected Honor and Awards

  • 2023 Google Ph.D. Fellowship in Mobile Computing
  • 2023 UbiComp Distinguished Paper Award 8 out of 210 accepted papers.
  • 2021 ACM MobiCom Student Travel Grants
  • 2020 ACM SenSys Best Poster Award Runner-ups 3 out of 84 accepted posters.
  • 2018 ACM MobiSys Student Travel Grants
  • 2015 ACM-ICPC Asia Regional Contest(Chang Chun Site), Gold Medal
  • 2015 CCF(China Computer Federation) Outstanding Undergraduate Award

    • Service

    Program Committee Member:
  • 2026 ACM UIST, ACM SenSys
  • 2025 ACM MobiSys, ACM SenSys
  • 2022 ACM SenSys Shadow PC
    • Associate Editor:
  • 2026 ACM IMWUT (UbiComp) Associate Editor
  • 2025 ACM IMWUT (UbiComp) Associate Editor
    • Reviewer:
  • 2025 ACM MobiSys, ACM SenSys, ACM IMWUT (UbiComp)
  • 2024 IEEE ICASSP, ACM IMWUT (UbiComp)
  • 2023 IEEE ICASSP, ACM TMC, ACM TOSN
  • 2022 IEEE ICASSP, ACM IMWUT (UbiComp), IEEE TMC, ACM SenSys Shadow PC
  • 2021 ACM TOSN, IEEE TMC, ACM MobileHCI
  • 2020 IEEE TMC, ACM IMWUT (UbiComp)
    • Organizer and Chair:
  • 2023 ICLR Machine Learning for IoT Workshop Co-organizer and Session Chair
    • Website designed by Ke Sun with OpenAI Codex.