In 2023, I read 87 papers and 26 non-technical books. 87 papers is slightly more than my previous record (82 papers in 2022), and I’ve never even been remotely close to reading 26 books in a year. Deciding to read more books is definitely…
Selected Publications
Controlling Vision-Language Models for Multi-Task Image Restoration
We present a degradation-aware vision-language model (DA-CLIP) as a multi-task framework for image restoration. DA-CLIP trains an additional controller that adapts the fixed CLIP image encoder to predict high-quality feature embeddings. By integrating the embedding into an image restoration network via cross-attention, we are able to pilot the model to learn a high-fidelity image reconstruction. The controller itself also outputs a degradation feature that matches the real corruptions of the input, yielding a natural classifier for different degradation types. Our approach advances state-of-the-art performance on both degradation-specific and unified image restoration tasks.
ICLR, 2024
How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?
We propose a benchmark for testing the reliability of regression uncertainty estimation methods under real-world distribution shifts. It consists of 8 image-based regression datasets with different types of challenging distribution shifts. We use our benchmark to evaluate many of the most common uncertainty estimation methods, as well as two state-of-the-art uncertainty scores from OOD detection. While methods are well calibrated when there is no distribution shift, they all become highly overconfident on many of the benchmark datasets. This uncovers important limitations of current uncertainty estimation methods, and our benchmark thus serves as a challenge to the research community.
TMLR, 2023
Image Restoration with Mean-Reverting Stochastic Differential Equations
We present a stochastic differential equation (SDE) approach for general-purpose image restoration. The key construction is a mean-reverting SDE that models the degradation process from high-quality image to low-quality counterpart. By simulating the corresponding reverse-time SDE, high-quality images can then be restored. We also propose a maximum likelihood objective that stabilizes the training and improves the restoration results. Our method achieves highly competitive performance on the tasks of image deraining, deblurring and denoising. The general applicability is further demonstrated via qualitative results on image super-resolution, inpainting and dehazing.
ICML, 2023
ECG-Based Electrolyte Prediction: Evaluating Regression and Probabilistic Methods
Imbalances of the electrolyte concentration levels in the body can lead to catastrophic consequences, but accurate and accessible measurements could improve patient outcomes. While blood tests provide accurate measurements, they are invasive and the laboratory analysis can be slow or inaccessible. In contrast, an ECG is a widely adopted tool which is quick and simple to acquire. However, the problem of estimating continuous electrolyte concentrations directly from ECGs is not well-studied. We therefore investigate if regression methods can be used for ECG-based prediction of electrolyte concentrations.
Preprint, 2022
Learning Proposals for Practical Energy-Based Regression
We derive an efficient and convenient objective that can be employed to train a parameterized distribution q(y|x; phi) by directly minimizing its KL divergence to a conditional EBM p(y|x; theta). We then employ the proposed objective to jointly learn an effective MDN proposal distribution during EBM training, thus addressing the main practical limitations of energy-based regression. Furthermore, we utilize our derived training objective to learn MDNs with a jointly trained energy-based teacher, consistently outperforming conventional MDN training on four real-world regression tasks within computer vision.
AISTATS, 2022
Accurate 3D Object Detection using Energy-Based Models
We apply energy-based models p(y|x; theta) to the task of 3D bounding box regression, extending the recent energy-based regression approach from 2D to 3D object detection. This is achieved by designing a differentiable pooling operator for 3D bounding boxes y, and adding an extra network branch to the state-of-the-art 3D object detector SA-SSD. We evaluate our proposed detector on the KITTI dataset and consistently outperform the SA-SSD baseline, demonstrating the potential of energy-based models for 3D object detection.
CVPR Workshops, 2021
How to Train Your Energy-Based Model for Regression
We propose a simple yet highly effective extension of noise contrastive estimation (NCE) to train energy-based models p(y|x; theta) for regression tasks. Our proposed method NCE+ can be understood as a direct generalization of NCE, accounting for noise in the annotation process of real-world datasets. We provide a detailed comparison of NCE+ and six popular methods from literature, the results of which suggest that NCE+ should be considered the go-to training method. We also apply NCE+ to the task of visual tracking, achieving state-of-the-art performance on five commonly used datasets. Notably, our tracker achieves 63.7% AUC on LaSOT and 78.7% Success on TrackingNet.
BMVC, 2020
Energy-Based Models for Deep Probabilistic Regression
We propose a general and conceptually simple regression method with a clear probabilistic interpretation. We create an energy-based model of the conditional target density p(y|x), using a deep neural network to predict the un-normalized density from the input-target pair (x,y). This model of p(y|x) is trained by directly minimizing the associated negative log-likelihood, approximated using Monte Carlo sampling. Notably, our model achieves a 2.2% AP improvement over Faster-RCNN for object detection on the COCO dataset, and sets a new state-of-the-art on visual tracking when applied for bounding box regression.
ECCV, 2020
Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision
We propose a comprehensive evaluation framework for scalable epistemic uncertainty estimation methods in deep learning. It is specifically designed to test the robustness required in real-world computer vision applications. We also apply our proposed framework to provide the first properly extensive and conclusive comparison of the two current state-of-the-art scalable methods: ensembling and MC-dropout. Our comparison demonstrates that ensembling consistently provides more reliable and practically useful uncertainty estimates.
CVPR Workshops, 2020
