Urban physical environments are the physical settings and built environments in neighbourhoods and cities which provide places for human activities. Evidence suggests that there are substantial associations between urban physical environments and various health outcomes, e.g. people’s physical activities might be influenced by surrounding physical environments, thereby affecting their health behaviours; more exposure to urban physical environments may benefit human mental health. Street view imagery enables us to capture the landscape at eye-level, making it a promising data source for observing and analysing the realistic dynamics of urban physical environments. Compared with traditional in-person assessments and field observations, street view imagery-based data collection is relatively time-effective and cost-effective. Researchers from epidemiology, psychology, and geography have used street view imagery to quantify the built environment and understand its impacts on public health. To summarize current research trends, this paper systematically reviews the use of street view images for sensing urban environments in public health studies. Specifically, we describe the characteristics of street view imagery and review the methodology for image processing and semantic understanding. We then summarize the challenges that remain for quantifying urban environments in terms of data and methodology. Several future research directions that would benefit public health research and practices are recommended in urban environment research.