Within & Between

It’s no mystery that research papers are essential for academic work. What IS a mystery is how to figure out who should be an author on those academic papers and how to have conversions about it. In this episode, Jess and Sara go over some authorship roles, guidelines and rules that govern authorship in the field, and talk about different ideas of determining order of authorship. This episode was recorded live at the Emerging Perspectives on Early STEM Learning, Development, and Education Workshop hosted by Purdue's Center for Early Learning.

In this episode we mention: 

In this episode, Jess and Sara reckon with what the preregistration revolution means in developmental science. We first visited preregistration and registered reports in Season 3. Since then, we’ve been a part of many preregistrations and registered reports, and have reviewed and handled them as editors. Through that process, we’ve noticed that questions often come up about whether something is really a preregistration and whether some studies or analyses can or should be preregistered. 

We mention: 

What can we say about the often used and often confused statistical approach that is moderation? Quite a lot actually. In this episode Jess and Sara introduce the basics of moderation (aka statistical interactions) using absolutely no math (or very nearly absolutely no math). Listen to get a primer on the kinds of questions you can ask with moderation, and to get answers to common moderation questions like: Why is my interaction negative if my main effect is positive? Can I interpret the man effect if the interaction is significant? Why is my interaction significant if my main effect isn’t? And how do I graph the result when I have continuous variables? 

(see the figure Sara is talking through on our website: http://www.withinandbetweenpod.com/)

Related to this podcast's interests in data sharing, this week Jess and Sara talk about common data elements initiatives. These are movements that are hoping to get all of the researchers in the same field or subfield to agree to a given set of assessments, measures, procedures, and/or reporting metrics (Think: Everyone who measures mother’s education would ask the same stem question with the same eight category responses, which would be coded and reported in the same way).  We’ll talk about different types of initiatives, the reasons why proponents think it’s a great idea, and what major concerns might come up. Sara also gives us a mini lecture introducing genome-wide association studies. Trust us, it’s related! 

In this episode, we talk about: 

Measuring change is our first ever LIVE EPISODE! Recorded at the Association for Psychological Science conference with a live conference audience! Over here in the developmental sciences, we are often trying to measure or predict how much people grow and change over time. To do that, we’ll sometimes measure a skill or ability twice (for kids, maybe that’s once near the beginning of the school year and once at the end of the school year). In this episode, Jess and Sara talk about options for statistically modeling and predicting change between two timepoints. We talk about two basic models: the simple difference scores and residualized gain scores, explain why we can’t just measure people at post test, and conclude that the question is important in making your decision. Have a listen as we get to take some related questions from the audience. 

Things we mentioned: 

If you are working in the latent space and have a measurement model at each time point, you may need to establish measurement invariance across time. To do so, you can follow steps in Brown (2015; pp. 259–265). 

Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). Guilford Publications.

Mediation models! At their core, mediation models explore the relations among three variables: A predictor, an outcome, and a secret third thing they call a Mediator. You’ve probably seen mediation models out there in the published literature, with folks testing paths, seeing if a relation is still there if a mediator is involved, or testing for an “indirect effect”. Jess and Sara talk about all of this in this episode. What is mediation, what kinds of questions can you ask with it, and why are people so mad about it? Jess and Sara go over the basics of this method, including a few examples, and talk a bit about the controversies surrounding it. 

One of the ways the research machine operates is through research conferences. Scientists present new work, hear about others’ work, and (maybe most importantly) network with one another. In the first of a two-part episode about conferencing, Jess and Sara talk about what research conferences are and why you might want to go. We talk through how we choose which conferences to attend, how to plan your travel, and the importance of a comfortable pair of shoes.

If a scientist learns something new about the world but doesn’t tell anyone, have they really learned something new? Telling people is precisely the point of a dissemination plan or broader impact statement that the Institute of Education Sciences or National Science Foundation asks for as part of their research grant applications. Federal agencies want you to share what you learn with the world. You might be used to thinking about this as publishing scientific papers or presenting your work at academic conferences, but that’s only one potential audience. These agencies also want you to share your work with the public. In this episode, Jess and Sara talk about creative and engaging ways to share your work with the public, including playing video games for science!

We talked about: 

Developing research questions is one of the most challenging aspects of the research process. This one little statement reflects so much about the rest of your project or paper. Jess and Sara reflect on how they develop their own research questions, the challenges with research questions and secondary data, and how they help teach students to develop research questions in their own work. Spoiler alert: Part of the answer is to read. A lot. And then read more.

We love data sharing around here, and something exciting happened In January 2023: The NIH put into effect a brand new Data Management and Data Sharing Policy. Any grant submitted to the NIH must follow this new Policy, and each grant submitted will include a Data Management and Sharing Plan describing how the research team will comply with that policy. Have a listen as Jess and Sara describe the new Policy, and walk you through the parts of the Plan, with honestly a bit more enthusiasm than should be allowed for reading and discussing a Policy.

We talked about:

Since we last talked, Jess has moved institutions! In this episode, Jess and Sara discuss the ins and outs of her recent move and why she made the decision to go. We describe how a mid-career academic move is different from an early career one, the timeline of the hiring process when you do (and don’t) have a promotion as part of the offer, and some of the difficult nuts and bolts of moving. Learning a new culture, new systems, and getting new logins. And did you know they delete your email when you move?

Some resources:

In most faculty jobs, the department or college asks you to stop and document what you’ve been up to for the past year. In this episode, Jess and Sara talk about the institution’s goals for this annual review. We compare notes on what the evaluation packet looks like for our two programs, and we find that they are… quite different. Finally, Sara has the inside scoop on what happens after you turn in the annual review packet at her institution, and folks it is fascinating. Have a listen as you’re prepping to do your year-in-review!

In this episode, Jess and Sara talk about managing all the manuscripts that they are working on. We talk about whether and how we keep track of those papers that are nearly done, partly done, or just an idea, and how that changes when you are a first author vs. a coauthor. We also talk about how we mentor students through the writing process, from idea through to finished draft. We’ll cover broad concepts and specific tools of the trade.

Mentioned in this article:

In this episode, Jess and Sara finally (Finally!) tackle the basic premise of behavior genetics. We talk about what kinds of research questions you can ask with genetically sensitive designs, describe what heritability is and what it’s not, and discuss just how it’s possible that scientists can use data from twin pairs to understand how much of the variance in some skill or behavior is due to genes and how much is attributable to the environment. A lot of that estimate depends on just how much genes and environments vary within a twin pair and between twin pairs (see what we did there?).

In this episode we talked about:

If you’ve ever had to deal with missing data, you’ve probably wished you could avoid it completely in the future. So why on earth would anyone design a study where data are missing on purpose? When you set up a project, there are actually several advantages to selecting a subset of people to skip assessments, items, or waves on purpose. Jess and Sara describe those designs here, and just what the advantages are, as well as fun future directions for one specific subtype of these designs: The two-method measurement planned missingness design. Even though we start with a basic missing data overview, and you might think that missing data is scary or boring, we SWEAR this one is interesting.

Links mentioned in this episode:

Why do you write a grant? To get money to do some sort of project of course! But how do you figure out how much money you need? Through a grant budget! But what goes into a grant budget? All kinds of things that you might expect (e.g., cost of the materials you need) and some things that you maybe don’t (e.g., Indirect costs). How can I find out what those budget-y things are and what they mean? You’re in luck, friend. That’s what Jess and Sara talk about for this whole episode! But Jess, why did you structure this whole show note blurb as questions and answers? Playing around with different narrative structures is both fun AND informative.

Preregistration is writing down your idea for how you plan to collect and analyze your data _before_ you actually start collecting or analyzing it. In this episode, Jess and Sara talk about the many benefits of preregistration, the differences between preregistration and registered reports, describe their experiences with it, discuss some resources, and (hopefully) put to rest some fears about the practice that you might have.

Some resources we mention in this episode:

Trajectory of heritability to the heritability of trajectories?

Measuring growth in skills and abilities over time is practically (maybe literally!) synonymous with developmental science. In this episode, Jess and Sara talk about how static measurements of a skill can have different variance, and different predictors, from growth in that skill. After a general discussion, we dive into how that plays into estimates of heritability, and the difference between the trajectory of heritability and the heritability of trajectories.

In this episode, Jess and Sara talk about what it’s like to serve on an academic search committee. To those of you who are about to serve on a committee (maybe for the first time), we will take you through the whole process from writing the call to making the offer. Or those of you on the market this year, peek behind the curtain.

Every search is different! Things happen very differently from university to university, and even from search to search. We want this overview to give you, the search committee member, the permission to speak up and suggest changes from the way it may have been done before.

In this episode, Jess asks Sara to talk about LDBase, the data repository she has been building designed specifically to hold developmental science data. We discuss what LDbase is, what motivated her and her colleagues to start building it, what makes it unique, and how it works. Sara even describes several excellent resources you can find on the website that make data sharing easier.

You may have noticed that the cycle of academic publishing is pretty broken. Scientists give their research papers to academic journals for free, then the journal puts that work behind a paywall. Sometimes institutions will pay for a package of subscriptions to these journals, often costing millions of dollars. Jess and Sara talk about this cycle, and how institutions, journals, and individuals have started to push back against it.

In this episode Jess and Sara talk about the role of methodologist in developmental science, and how it’s different from a statistician. A methodologist is a content expert and a collaborator like any other, and we do our best work when we’re involved right from the initial planning stages of a project. Why? Because nearly every decision made in a research project can have implications for statistical conclusions, and methodologists are the collaborators who keep this in mind. Think you might be interested in being a methodologist? Jess and Sara give some suggestions on how to prepare and frame your materials.

Resources:

Well the spring 2021 semester is over, and - what is this feeling? We’re a little (a lot) overwhelmed. Jess and Sara describe their experiences, and theorize a bit about why it might be particularly bad right now. Is the pandemic over here in the US? What does that mean for our jobs? That feeling of one day more can sometimes be too much. We end by suggesting, maybe, we all need a break, and to give ourselves, and our colleagues, some grace.

The saga continues. In this episode we talk about the data management steps you’ll need to consider after you’ve collected your data. We talk a lot about data entry, including the science of how to do it well. We also discuss data merging, data cleaning, and the concept of “release” datasets. Finally, because it seems no episode is complete without Sara discovering something to be shocked about regarding Jess’s approach to science, we talk about the importance of backing up your data.

Resources:

You might think “data management” is just getting data ready to share, but it’s so much more. In this episode, Jess and Sara talk about the expanded universe of data management. This is the first of a two part series, and focuses on some of the things you can do to help ensure you have good quality data before you even talk to a single participant. We talk about lots of fun data management tips and tricks, from creating research protocols, assigning IDs, creating variable names, variable codes, and coding missing data. Make a rule. Write it down. Don’t change it.

Mentioned in this episode:

Turnabout is fair play in this episode as now Sara describes her new paper! Some developmental science research is focused on how the home environment is correlated (or co-occurs with) children’s skills, behaviors, or abilities. Sara explains that these correlations are also influenced by the shared genes between parents and their children, and that the genetic correlation may be inflating the true environmental correlation that scientists are seeking to measure. In this episode, she describes a few different ways that researchers can account for shared genetic influences in their work. The solutions range from very simple to very complex, but all are very exciting.

New Paper Alert! Jess and her colleagues have a paper in review about a particular statistical method that we see a lot in developmental science. Have you ever run a confirmatory factor analysis and then saved (or extracted) the factor scores, turning them into observed scores for use in another analysis? If you have more than one latent factor, the default method for extracting factor scores can have some unintended consequences on the correlations between those newly created observed variables. But also describes how you can correct it. The link to the paper gives you the code to do it yourself. This episode is a bit numbery and a bit technical, but we think relevant for developmental scientists!

Note. Posted with apologies to our reading research friends. Jess did not come up with good examples of what comprises reading on the fly in this episode. I am well aware a good fonging is in order. I do know better. I am sorry.

To celebrate hitting our 10,000 listens benchmark, Jess and Sara spend episode 10 discussing two listener questions. First, we talk about the decision to change or not to change your last name when you get married. It’s something many developmental scientists struggle with, and we talk about how (even though it feels unique) it’s not terribly different from other professions. Second, we discuss that overwhelming feeling you sometimes get when you’re trying to manage your research pipeline, and how we (try) to (mostly) keep it at bay.

Ps, how do you say ORCID?

Are you open-science curious? Ready to turn and face the strange ch-ch-changes happening in our field? Preprints are one of the most accessible steps in learning about, and might we even suggest embracing open science. To post a preprint, you put a digital copy of a completed paper online, somewhere others can read it.  In this episode, we discuss the advantages of preprints, different ways to post them, and address some of the potential barriers and common fears we hear from other scholars  when we talk about preprints.

Jess and Sara are both advocates for interdisciplinary science, and interdisciplinary science means collaborations! On this episode, we talk about why we love interdisciplinary science and why some departments would prefer a more focused line of research. We also talk about the life cycle of collaborations, how we start them, maintain them, and let them go.

Links from this episode:

New year, new goals! Goal setting helps keep you on task, helps you work towards bigger goals, and can help you feel more successful. But many of us really struggle with it (cough cough Jess cough cough). Sara walks us through the goal setting system she uses, how it differs by career stage. We talk about specifics including what to include in your long term goals (hint, only the things you can control), and how to set up a semester spreadsheet.

In this episode:

What does it mean to have a lab, and who is part of it? In this episode, Jess and Sara describe their experiences with different lab setups and structures, and how they manage their meetings with students, projects, and collaborators. We discover that Sara has been using a lab management and productivity tool secretly for years! Our systems for working with our labs, projects, and students are always changing, and Sara might be a bit too addicted with trying to find the best combination.

Links from this episode:

As the semester draws to a close, Jess wakes up in the middle of the night concerned not with completing her grading, but with measuring inter-rater reliability for coding schemes. Reliability can refer to the agreement among raters, or agreement among items. There’s a lot of online discourse right now around reliability, but most of it is focused around the former: agreement among items. Over here in the field of developmental science, we’re often coding human behavior through observations, including implementation fidelity. With scales like these, reliability is used to mean several different types of agreement among raters: Training coders to a reliability standard, “in-field” reliability between pairs of raters, “drift check” reliability as you continue to collect data over time, and then overall or average inter-rater reliability for all raters and observations. Only after assessing all of these can you begin to examine reliability designed to assess agreement among items.

Papers to read about establishing and measuring inter-rater reliability:

Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23.

Bruton, A., Conway, J. H., & Holgate, S. T. (2000). Reliability: what is it, and how is it measured?. Physiotherapy, 86(2), 94-99.

Also mentioned

McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412.

Raykov, T., & Marcoulides, G. A. (2019). Thanks coefficient alpha, we still need you!. Educational and Psychological Measurement, 79(1), 200-210.

Edwards, A., Joyner, K., & Schatschneider, C. (2019, June 27). A Simulation Study on the Performance of Different Reliability Estimation Methods. PsyArXiv. https://doi.org/10.31234/osf.io/xzc52.

It is a truth universally acknowledged that an academic in 2020 must be in want of an online presence. Jess and Sara describe why you need to be online, and their experiences with social media, professional networks, and creating websites (and that twitch isn’t just what happens to Jess’s eye after 6 straight hours on zoom calls). The episode ends with our top three suggestions for how to get involved online.

Links from this episode:

To be or not to be (a parent!) that is the question. One of our listeners reached out with a question about timing of starting a family during an academic career. Jess and Sara both describe their respective experiences with having children, how quickly they got back to work, and the pros and cons of the academic work-life balance.

In which we welcome you back to a new season of the podcast. Jess and Sara catch up on pandemic work and life. We describe the subtle and sometimes not so subtle feeling of being underwater that seems to accompany this particular time, describe how our online teaching is going, and discuss the different ways we’re translating and incorporating anti-racism into our courses.

There’s a close link between the concepts of Perfection and Impostor Syndrome, perhaps mediated (or moderated?) by experiences of rejection. In this episode, we discuss that link, and personify it as Sara helps Jess try to cope with rejection and encourages her to get some rejected manuscripts back under review*.

In this episode we talk about rejections - paper rejections. Jess and Sara describe their emotional and not so emotional reactions to finding out whether a paper has been rejected, offered a revise and resubmit, or accepted. We also discuss how much we change a rejected manuscript before we send it elsewhere, how to craft a revision letter, and the urban legend of  papers accepted on the first submission.

In this episode, we revisit how we’re working during COVID. Topics include excellent desk space (Sara) or lack thereof (Jess), and how we are and are not balancing parenting and working at the same time. We also talk about preparing to teach at a distance, and whether we’re even allowed back on campus.

Peer review is, for now, an essential part of the scientific process. Writing a good peer review is not easy, and we feel it’s rarely formally taught. We get no feedback on the process, so it’s difficult to learn. Listen to Jess and Sara talk about how they decide when to say yes to a review (and when to say no), how to schedule them into your work week so you don’t lose track, and what your job is as the reviewer. We also give some suggestions on how to get started reviewing, and how to ask to be on an editorial board.

Grant information continues! This time from the reviewing side. Jess and Sara talk about the process of reviewing grants for the big three US federal agencies. The grant reviewing process is an elaborate, multi-step process, which we will outline and discuss. Listen to this if you’re curious what kinds of things your reviewers are doing when they get a stack of grants to review, or if you’re new to serving on a grant panel and want a preview into what it’s like to be in the room. 

Writing is one of the loneliest activities. For some of us, it can be hard to keep yourself accountable, especially right now with not much of a regular schedule and lots of working from home. On this episode we talk about our paper writing process, how we go from an idea to a submission, and the wonders of co-writing. We are divided on the use of the phrase “dot-jot notes”, which google says is NOT a Canadian only thing.

It’s grant season! In this episode Jess and Sara talk about the way they approach writing grants. There are many different kinds of grants, so in this first grant-focused episode we briefly discuss the different funders for developmental science research, how to choose a grant mechanism to apply for, the general rules and process that we follow for any grant, and the critical role of ice cream.

This week, we tackle the history of racism in our specific developmental science fields. We believe that Black Lives Matter. We believe it’s our job to do the anti-racist work within our own departments and fields. In thinking about actionable steps we can each take to work be anti-racist in the academy, we first acknowledge that we are two white women who can not speak to the lived experiences of Black, Indigenous and People of Color in the academy. See the rest of the show notes for excellent readings and voices to listen to about how to do anti-racist work.

In this episode, we use our platform to shine light on what may be hidden to many researchers in developmental science, the white-supremacist history that underlies both behavior genetics and statistics. Increasing our own understanding and sharing the history of these fields is part of our own anti-racist work. We hope this episode will encourage you to start to study the history of your own specific subsection of the field.

Some of our favorite readings and a listening list from this episode:

Jess and Sara continue the COVID discussion, talking about ideas for alternative ways to collect data during the 2020/2021 school year, the reasons NOT to collect data next year, and some alternative ideas for your research. For those of you who do intervention work, we also discuss some analytic options you have if, due to school closures or forced research lockdowns, you suddenly find yourself in a pre-test only design situation.