The header image above was generated using the text-to-image tool at imagine.art, from the input “teddy bear scientists having an intense discussion in a reading group about machine learning”.
Since I started my PhD almost five years ago (in September 2018), I have categorized, annotated and written short comments for all research papers I read in detail. I share this publicly in a GitHub repository, and recently reached 300 read papers. To mark this milestone, I decided to share some thoughts on why I think it’s important to read a lot of papers, and how I organize my reading. I also compiled some paper statistics, along with a list of 30 papers that I found particularly interesting and/or well-written.
Why I read a lot of papers
The simple reason as to why I try to read a lot of papers, is that I find it very enjoyable and interesting. Among all the different everyday activities of a typical PhD student, reading papers is one of my clear favorites. However, I also think it’s key for becoming a good researcher. In fact, I consider reading papers to be a fundamental part of what it even means to be a researcher.
First of all, I have found it to be a very powerful tool for learning new things. It’s actually quite remarkable how much you can learn about a certain machine learning problem or method simply by reading e.g. 5 papers about it in detail.
Secondly, I think it’s a great way to generate new research ideas. When reading an interesting paper, I virtually always get at least one idea about how the proposed approach potentially could be extended in some way, or how it could be combined with some of my previous work.
How I’m able to read a lot of papers
As a PhD student, I’m always quite busy with research, teaching duties, coursework and other meetings. In my experience, the single most important thing one can do in order to read a lot of papers is therefore to actively prioritize reading. This means setting aside some time in your schedule each week, specifically for reading. Aiming to read at least one paper a week has generally worked well for me, and I thus make sure to schedule at least one 1-2 hour time slot each week. By treating reading just like any other weekly meeting, I have (almost) always been able to find time for it.
The second most important thing one can do is probably to join (or start) a reading group. I’ve been organizing a weekly machine learning reading group at our division since I started as a PhD student. Being a member of a reading group makes it much easier to actually set aside time to read e.g. one paper a week.
Joining a reading group also comes with many other benefits. First of all, it’s a good way to “force” yourself to read papers you wouldn’t necessarily have selected yourself (such papers can be surprisingly interesting). I think this is important, to not just read the very latest state-of-the-art papers within your specific area of interest. Instead, I think one should actively try to branch out a little and also read some older papers, and papers from other areas.
Moreover, discussing a paper you’ve read in a reading group almost always significantly improves your understanding of the paper. Joining a reading group is also a great way to learn more about your colleagues’ research and interests, and of course, discussing papers with other people is a generally fun and enjoyable activity in itself.
How I organize my paper reading
For each paper I read in detail, I create a note somewhere where I can easily find it later. In this note, I write down any questions, thoughts or ideas which arise during reading. Reading papers can generate a lot of research ideas, and they all seem obvious right there and then, but if you don’t write them down they will be very difficult to remember in the future. A PhD is long, you might want to go back to a certain read paper years later, and then such notes (even if they are very brief) can be incredibly useful.
Afterwards, I also write a very short summary of the paper, quickly answering questions such as “Was the paper interesting overall?”, “Was it easy to understand?” or “Could it be relevant for my research now or in the future?”. Having short answers to simple questions like these can also turn out to be surprisingly helpful.
For each paper, I also annotate the pdf and upload it to my GitHub repository (i.e., somewhere where I can easily find it later). I strongly recommend getting a tablet for reading and annotating papers, it really does make a big difference.
The main reason for why I annotate each paper is that this makes it easier to stay focused while reading. I find that actively annotating a paper “forces” you to actually try to understand what you’re reading. Moreover, it enables you to go back to a paper years later and quickly find the most interesting and important information.
Why I share my reading publicly
The reason why I share a list of all my read papers publicly on GitHub is somewhat of a coincidence, it’s mostly just an idea I happened to have one day when I started my PhD. Now almost five years later, it is however something that I definitely can recommend.
There are of course much more advanced tools such as Zotero (which I’m sure can be super useful and convenient), but I really enjoy the simplicity of my “GitHub repository” system, and it seems to provide all the features I need. For example, it enables me to go back and very quickly find particular papers that I’ve read before.
Moreover, I strongly believe in the general principles of open science. Open exchange and publication of information are defining features of the scientific community, and publicly sharing all of my reading in this way is, at least to me, a natural next step towards truly transparent and accessible research.
Last but not least, publicly sharing my reading definitely motivates me to read more papers. It adds a bit of external pressure and encouragement to stay consistent with my reading, to actually set aside some time (almost) every week. Without it, I’m quite confident that I wouldn’t have been able to reach 300 read papers already.
How I find interesting papers to read
To read a lot of papers, one also has to find a lot of interesting papers. I regularly look for interesting new and old papers, for example by checking:
- arXiv.
- Twitter.
- Accepted papers lists for upcoming and previous conferences.
- The references in interesting papers.
- The “Cited by” list on Google Scholar for interesting papers.
- A list of people whose research I find particularly interesting, occasionally checking arXiv/scholar for their new papers.
I have a (nearly) daily habit of quickly going through all new cs.LG and cs.CV papers on arXiv, and spending 5 minutes scrolling through twitter. The number of papers on arXiv keeps increasing every year, but I still find it manageable and worthwhile to go through them daily (typically, this takes at most 20-30 min). There are many aspects of twitter that I really don’t like (thus I limit myself to 5 min per day), but I must admit that it’s a good tool for finding interesting papers, researchers, workshops, seminars etc.
Whenever I find a seemingly interesting paper, I go through it very quickly (check the abstract, scroll through the method and/or experiments) and decide whether or not it still seems interesting. If so, I then save it somewhere where I easily can go back and find it later. Specifically, I put the paper titles in Google Keep lists, and if a paper seems especially interesting I also make a short note about that. Once I have time to read a paper in detail, I then go through my lists of interesting papers and pick one.
30 Particularly Interesting and/or Well-Written Papers
I went through my comments for all 300 read papers, and selected a list of 30 papers that I found particularly interesting and/or well-written:
Uncertainty Estimation:
- Weight Uncertainty in Neural Networks (ICML 2015)
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning (ICLR 2020)
- Laplace Redux – Effortless Bayesian Deep Learning (NeurIPS 2021)
- Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow (ECCV 2018)
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (NeurIPS 2018)
- Selective Classification for Deep Neural Networks (NeurIPS 2017)
- Conformalized Quantile Regression (NeurIPS 2019)
- On the Practicality of Deterministic Epistemic Uncertainty (ICML 2022)
Out-of-Distribution Detection:
- Hierarchical VAEs Know What They Don’t Know (ICML 2021)
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks (NeurIPS 2018)
- SSD: A Unified Framework for Self-Supervised Outlier Detection (ICLR 2021)
- Out-of-Distribution Detection with Deep Nearest Neighbors (ICML 2022)
- Being a Bit Frequentist Improves Bayesian Neural Networks (AISTATS 2022)
- Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection (NeurIPS 2021)
- Does Your Dermatology Classifier Know What It Doesn’t Know? Detecting the Long-Tail of Unseen Conditions (Medical Image Analysis, 2022)
Energy-Based Models:
- Estimation of Non-Normalized Statistical Models by Score Matching (JMLR, 2005)
- Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models (AISTATS 2010)
- On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models (AAAI 2020)
- Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (ICLR 2020)
- Flow Contrastive Estimation of Energy-Based Models (CVPR 2020)
- Joint Training of Variational Auto-Encoder and Latent Energy-Based Model (CVPR 2020)
Other:
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020)
- Neural Unsigned Distance Fields for Implicit Function Learning (NeurIPS 2020)
- Expressive Body Capture: 3D Hands, Face, and Body from a Single Image (CVPR 2019)
- A Baseline for 3D Multi-Object Tracking (IROS 2020)
- Learning Latent Dynamics for Planning from Pixels (ICML 2019)
- Blind Spots in AI Ethics (AI and Ethics, 2022)
- The Uselessness of AI Ethics (AI and Ethics, 2022)
- Why AI is Harder Than We Think (GECCO 2021)
- Talking About Large Language Models (arXiv, 2022)
Paper Statistics
I compiled some more detailed statistics of my read papers (number of papers read each year, papers by publication year, papers by venue, papers by category):
303 read papers.
1.84 GB of annotated pdfs.
