In the News, Stanford Data Science Scholars Program

Intellectual Curiosity Meets Machine Learning: Rylan Schaeffer’s Journey into Data Science

Beginnings with A Meandering Path

Currently a Stanford Data Science scholar, Rylan Schaeffer’s interest in data science developed during his undergraduate years. Unsure of what degree to pursue, Rylan started as a biomedical engineering major but quickly realized that his memory was too poor for biology. Next, he considered economics and business and became the Chief Financial officer overseeing 30 business units and the $10 million Student Government budget, but the courses felt insufficiently challenging. His university, UC Davis, offered a unique opportunity to design an individualized major. He crafted a program focused on persuasion, integrating elements of consumer science, communication, sociology, and anthropology, but found it lacking rigor. He ultimately settled on double majoring in Computer Science Engineering and Computational Statistics. In his final quarters, he enrolled in courses on machine learning and artificial intelligence just as Google DeepMind’s AlphaGo dethroned Lee Sedol at Go. The novelty and success of the algorithms fascinated Rylan, beginning his career in data science, Neuroscience-inspired AI, and Machine Learning.

Rylan pursued a master's in Cognitive Neuroscience at University College London, researching metacognition. Afterward, he joined Uber as a data scientist for a year before returning to academia to complete a second master’s in computer science at Harvard, then interned at Google DeepMind before beginning his PhD. This winding path has shaped his professional trajectory.

An Award-Winning Paper: Investigating Predictability and Surprise in Large AI Models

In the realm of artificial intelligence, the task of understanding large-scale models—often referred to as “foundation” or “frontier models”—is becoming increasingly complex yet essential. “Frontier or foundation models are large-scale AI models trained on vast datasets using immense computational power,” explains Rylan. “Their purpose ranges from understanding language to complex problem-solving.” Rylan’s recent research is focused on grasping not only the capabilities of these models but also their limitations to ensure they are more predictable and trustworthy.

One primary goal of his research is to develop scaling-predictable and human-interpretable evaluations of such frontier AI systems. Historically, the field had discovered that as models grew larger in terms of their parameters, their training data, or the total computational operations, the models would display consistent, predictable improvements. However, recent work discovered the so-called “emergent abilities” of such models, whereby models exhibit seemingly unpredictable leaps in capabilities.

In a research paper titled “Are Emergent Abilities of Large Language Models a Mirage?” Rylan investigated whether these emergent abilities were truly the result of fundamental changes within the models or rather due to the way researchers approached their analyses. Through various tests, he found that emergent abilities could appear or vanish depending on how the data was analyzed. In other words, models didn’t necessarily exhibit new skills spontaneously; rather, the methods used to measure these abilities created the illusion of abrupt changes. This paper, with Brando Miranda and Rylan’s faculty advisor Sanmi Koyejo, received the Best Paper award at NeurIPS 2023.

Parallel Between Artificial and Biological Systems

Before working on frontier AI systems, Rylan’s research focused on neuroscience-inspired artificial intelligence (NeuroAI). One area he focused on was understanding the origins and advantages of unique Nobel-prize-winning neural representations found within the mammalian lineage that play a crucial role in spatial problem-solving.

Mammals rely on specific neurons to navigate and interact with their environments. For instance, to find food, return home, or travel to familiar locations, they must solve fundamental spatial problems:

Localization: determining one’s position
Mapping: creating a mental map of the environment
Navigation: charting a path from one point to another

These spatial problems, which humans solve effortlessly, are supported by specialized neural representations in the mammalian brain. NeuroAI has long been interested in understanding the optimization processes that produce these unique neural representations, including notable contributions from Google DeepMind and spotlighted studies presented at NeurIPS, the premier machine learning conference. These studies sought to uncover the origin of these neural representations. Significantly, he and his collaborators made fundamental contributions to understand under what conditions, and why, artificial neural networks learn neural representations matching biological neural networks. This discovery highlights a fascinating parallel between artificial and biological systems in their methods for solving spatial challenges.

Advancing Research and Fostering Collaboration with Stanford Data Science

“The Stanford Data Science program has become an instrumental resource for researchers and students, facilitating a vibrant community where knowledge exchange and collaboration thrive,” says Rylan. “For students and faculty members, the program provides both a strategic foundation for long-term projects and short-term, tactical support for specific research challenges.”

According to Rylan, one of the standout benefits of the Stanford Data Science program is its strong community of scholars, fellows, and faculty mentors. For many students, departments can feel isolated due to their specialized focus. By joining the Data Science program, the students gain access to a diverse range of experts from across the university, making it easier to break down these silos. For instance, computer science students often remain focused in their building or lab, limiting cross-disciplinary interactions. The data science community, however, bridges these gaps, fostering interdisciplinary connections that spark new research ideas and offer fresh perspectives.

For example, Rylan described how connecting with a political science expert within the program helped him design a machine-learning project on political campaigning. This connection allowed him to receive expert feedback on his project idea, refining the approach and identifying valuable areas of focus. Such exchanges are part of what makes the Data Science program invaluable—participants can step out of their silos, meet researchers from different fields, and share ideas.

Laying the Groundwork for Future Collaborations

“Beyond just meeting new people, the program sets the stage for meaningful, long-term collaborations,” Rylan emphasized. “Through regular interactions, scholars build rapport with others who may become future collaborators on research projects. Whether it's exploring new methodologies or tackling societal challenges, these relationships lay a strategic foundation for research that could span years.”

In addition to fostering collaboration, the Stanford Data Science community offers real-time solutions for researchers who are navigating technical hurdles, often through the program’s Slack channel. When students or faculty face issues, they can post a question to the community and often receive a solution the same day. This kind of assistance is particularly valuable for scholars working under tight deadlines or those needing guidance on unfamiliar topics.

For example, another data science scholar recently needed to run machine learning experiments on a tight deadline. He was initially informed that his computations could take a month, which was far too long. However, by consulting with Rylan and the data science community, he received immediate feedback and suggestions for optimizing his work. Community members helped him troubleshoot GPU utilization, bottlenecks, and alternative computing platforms. This prompt support enabled him to finish his experiments on time, illustrating how the program’s resources can help scholars stay on track and meet pressing deadlines.

How Has Data Science Changed the Field

“Honestly, I’m not sure I have a definitive answer to this question,” admits Rylan. “In some ways, data science has completely transformed the field of machine learning—and in other ways, it feels like nothing has changed.” Rylan goes on to explain that machine learning always has been about data: “We’ve always collected large datasets, trained models on them, and refined our models based on the results. The general workflow hasn’t changed: we still worry about data quality, about feeding the right data to the right model, and about how optimization aligns everything. In that sense, the core of what we do hasn’t shifted drastically.”

Rylan continued, “Yet, in other ways, the field has evolved immensely. We’ve become far more rigorous, using advanced data science and statistical tools to iterate on our models, optimizers, and data. Now, it’s common for researchers to analyze datasets deeply, asking questions like What’s actually in this data? When is it useful? What does it do well—and what doesn’t it? Similarly, we now study the behavior of the models as they learn, tracking how they evolve and trying to understand the internal dynamics. So, while the overarching process remains the same, our approach has become much more meticulous and sophisticated.”

“On the plus side, doing these analyses today is more affordable and accessible because of major improvements in data handling and model training efficiency. Large datasets are better cleaned and understood, and we have parallel pipelines to analyze them. But there’s a flip side: everything has gotten much larger and more complex. In the past, we worked with smaller datasets, like MNIST (a classic set of handwritten digits). Then we moved to datasets with millions of samples, like Professor Fei Fei Li’s ImageNet, and now we’re training language models on trillions of tokens. Similarly, models themselves have grown from simple structures to “deep” architectures, to today’s massive models that require multiple data centers to grow. It’s a double-edged sword—the tools are more powerful, but the scale of what we’re tackling is enormous,” Rylan explained.

He further explained, “Even optimization techniques have become more complex. We used to rely on basic gradient descent; now, we use more sophisticated methods that demand significantly more computational resources. In every respect, things are growing larger and more intricate.”

The Impact of Data Science on Academic Research

In Rylan’s opinion, data science has made a big impact on academic research. For one, the quality of tools available has greatly improved, making it easier to produce high-quality analyses and visualizations. This shift has raised the standard for research papers. Papers that might have seemed groundbreaking a decade ago might not even get accepted now because expectations have evolved.

The other major impact is on research speed. With better tools and coding assistants like Copilot, as well as chatbots for reasoning through ideas, it’s now much faster to run experiments and analyze data. However, the increased pace comes with risks. Mistakes are easier to make, and sometimes even sophisticated analyses might turn out to be flawed—so polished they look correct, yet fundamentally misguided. It’s both an advantage and a challenge. The field benefits from the higher quality and volume of research, but there’s also a risk of getting lost in complexity, with errors slipping through because they’re harder to detect.

Passions Outside of Data Science

Outside his research, Rylan’s favorite quarterly activity is organizing and co-hosting pastry crawls through San Francisco with Stanford Data Science alumnus Ariana Mann. Rylan has hosted so many he can now spot the novices because they arrive hungry, eat too much at the first stop, and end up too full to enjoy the rest. The veterans know to pace themselves, taking small bites and sharing with others, to last through all the stops.

Outside of food adventures, Rylan loves spending time with friends, exercising, swimming, reading, and traveling to incredible places for machine learning conferences. He also has a beautiful Miniature Australian Shepherd named Trina Louise (photos), who Rylan describes as “terrified of life—I’ve never seen a creature so scared of existence.” Trina is a rescue dog, and Rylan wishes he knew her full backstory to understand her better and to help her more.

Advice for His Younger Self

“I love this question because there’s a piece of advice I was given that has stuck with me,” says Rylan. Before he joined Stanford, he spoke to a Computer Science PhD student who told Rylan that the most important characteristic for being a successful researcher isn’t being the smartest or the hardest working, or even the most creative, but accelerating when the work becomes miserable. Every project, Rylan says, inevitably has extremely unpleasant parts, and reminding himself of this simple truth has helped him tremendously. “That’s been incredibly true for me, and I think it’s a valuable lesson for everyone.”

Rylan has also been collecting “advice” over the years from books, movies, and conversations. He even has a section on his website where he displays random quotes that have resonated with him. “Different things speak to us at different times in life, but I find it meaningful to keep these snippets of wisdom around for when I need them.”