Melissa Pappas
Rigor in how science is done and how results are analyzed and communicated is essential for science to be useful in the real world. Without it, significant results become meaningless and trust in the scientific community diminishes. Also, a lack of rigor not only increases the spread of misinformation, it can severely damage systems that people rely on to make informed decisions. To uphold a high standard across disciplines, a group of scientists from institutions across the country, including Penn Engineering, came together to create C4R, the Community for Rigor.
With funding from the Initiative to Improve Education in the Principles of Rigorous Research, a grant program of the National Institutes of Health’s (NIH) National Institute of Neurological Disorders and Stroke (NINDS), C4R was founded in 2022. The specific grant, “Creating an Educational Nexus for Training in Experimental Rigor,” or CENTER, is led by Konrad Kording, Nathan Francis Mossell University Professor in Bioengineering at Penn Engineering and in Neuroscience at Penn Medicine, and includes a collaborative community of faculty and researchers from ten institutions, bringing together a diverse set of research expertise. Their mission? To teach the principles and practices of scientific rigor to a broad audience and create more credible, collaborative research.
To achieve their mission, C4R is creating a free, online and open educational resource as a full curriculum focused specifically on scientific rigor topics such as good experimental design, performing proper statistical analysis and more.
“There are many steps to the scientific method and problems occur at every step,” says Kording. “We all need to work together to improve how we step through the method — we need to work towards a culture of improving science.”
To facilitate this intentional shift, Damon Centola, Elihu Katz Professor of Communication, Sociology, and Engineering with appointments in the Annenberg School for Communication and in Penn Engineering, leverages his research expertise in social networks and behavioral change as a scientific advisor for C4R.
“I am always looking to improve the rigor of the methods used in the social sciences,” says Centola. “I spend years setting up each experiment to ensure my data is not biased and represents a realistic demographic relevant to my research question. To ensure replication, I run multiple independent experimental studies with unique groups to identify whether the collective dynamics observed in one group are reproduced across independent populations. These methods generate results that have concrete implications for real-world problems. I train all my students in this mindset to help future scientists pursue the development of new, increasingly rigorous methods.”
Centola then uses his research on networks to inform C4R and integrate a new standard of scientific rigor into the culture of the entire scientific community.
“Movements and societal norms manifest once they reach a tipping point that allows information to spread and integrate into our culture,” he says. “My work aims to identify what kind of network structure facilitates a change in the way that information is used in society. I’m helping C4R develop that network structure to reshape the culture of scientific rigor within university research centers.”
In September, Centola and other members of C4R gathered to discuss their research and how they plan to integrate their work into the mission of C4R and vice versa at their first annual symposium.
“Our first annual meeting was successful,” says Carolina Garcia, Community Engagement and Communications Lead at C4R. “We hosted over fifty experts, students and collaborators from more than twenty institutions across the U.S., and held workshop sessions to test our educational materials, panel discussions about culture and behavior change, as well as firsthand accounts of graduate students’ experiences, and networking and community-building activities. We’re looking forward to expanding our annual meeting each year to become the world’s main scientific rigor gathering, the Community for Rigor Conference.”
In addition to a strong community of researchers, educators and innovators, C4R provides a free, open and online library of co-created units or mini courses on various topics in the realm of rigor, having just launched their first four units at the Society for Neuroscience conference in October.
“To make rigor more accessible, we create units on topics that help researchers understand where they may be making unintentional and biased mistakes in their work, from developing a research question to analyzing results and making conclusions,” says Kording.
“One of the engineering disciplines where rigor is extremely important is machine learning,” he continues. “If we want to diagnose mood or diseases based on phone or video data, we need to make sure we are collecting and analyzing data as rigorously as possible to keep this technology safe and useful. A few years ago we published two papers showing how popular approaches in medical machine learning produce false optimism, an unintentional consequence that we hope to help researchers avoid with the units and resources that come out of C4R.”
The initial units, an Introduction to Rigor, Causation Versus Correlation, Formulating a Valid Research Question and Improving Research Rigor With Randomization, were co-created from inception to publication, and the community welcomes anyone who would like to offer ideas for topics to be covered in future units. In addition to presenting these units at the conference, Kording hosted a conference workshop session, Community for Rigor: A Tool for Teaching the Principles of Rigorous Research, with five other C4R members presenting their work.
“We wanted to get our tools in front of the very audience we hope to attract to C4R,” says Kording. “And it worked out really well. People left the workshop with a better understanding of rigor and also with an understanding of the materials they can expect from us.”
Two successful conferences and four units of rigor later, C4R has come a long way since its inception. Now the community is looking forward to the launch of their new website and getting a jump-start on their next set of educational units to expand their work even further.
Visit the C4R website to contribute ideas, access their free materials and join the community.