Several months ago, I finally got my complicated research system to work.
It started in Prolific. Participants moved from there into Qualtrics, from Qualtrics into an embedded interaction with the OpenAI API through FastAPI, back into Qualtrics, and then back to Prolific. While all that was going on, the system collected snapshots across a creativity task, saved transcripts of participant-AI interaction, backed them up, kept track of what belonged to who, and produced structured data that was ready for analysis and went into encrypted storage. (An earlier version also included a think-aloud component with its own question/answer workflow.)
This might surprise you, but none of this worked immediately.
I ran into authentication problems, formatting issues, data going all over the place, things working locally but not everywhere they needed to. Participant paths had to survive refreshes, and failures, and people in general. I had to preserve a lot of transcripts without accidentally associating them with the wrong participant (or question, or condition). I can't even tell you about all the problems because I fixed them and moved on.
So, eventually it all worked together the way it was supposed to. It was an exciting time. I was running pilots and studies, collecting data, and had something to show my advisors that I was still working. And then.
And then, I realized that I might have built the wrong study, and therefore the wrong infrastructure.
Not wrong like the system malfunctioned. Oh no, that's what we would expect. But actually the system was getting better at doing exactly what I designed it to do. The problem was that I didn't think what I designed it to do actually measured what I wanted to understand. (Yay!)
It sounds kind of obvious when I lay it out in a couple of sentences like that. Of course the technical implementation of a study would be subordinate to the research question, and a functioning data pipeline doesn't actually make a measure valid, and when you discover the data isn't aligned with your questions, of course you redesign the study.
It did feel far less obvious at the end of a project it took months to make work. There's some invisible work between "I have a study idea" and "I'm ready to collect data".
For computational and AI-mediated research, that can often look suspiciously like software engineering. I built interfaces, connected systems, learned APIs, did troubleshooting, worked out what to log and how, discovered all kinds of fun edge cases. I fixed one problem only to create another (I was lucky if it was just one). Finally got to focus on testing the participant experience (and the researcher experience through structured outputs that prevented future me (present me?) from having to reconstruct the experiment from a folder with a bunch of CSVs with _stuff as part of their names.
For me, this definitely started feeling like progress on its own. And it was progress. Unfortunately, I couldn't use the fact that it worked the way I wanted it to as evidence that the study worked. It was a bit harder to maintain that distinction after many hours of coding and fixing and coding, etc.
Every time I solved a technical problem, it felt like the study implementation was that much more concrete. I was starting to have a real study like a real researcher. I had a URL, and participants, and database entries, and transcripts, and variables with names that made sense. I ran the study through from beginning to end and watched the data appear WHERE IT WAS SUPPOSED TO APPEAR.
So now, changing the study feels bigger than my usual new idea. It feels like I am taking apart all that hard work. And I had to get over that (and I'm sure I will someday).
Obviously I immediately moved into "what existing infrastructure can I preserve?" mode. What if I just modify it? Will another measure somehow fix this? Maybe I can fix it in the instructions, or even keep the architecture, but kind of reinterpret what I was doing.
Some of those ideas were legitimately reasonable, even to other people. But then I started thinking about how I could make the research question fit this system I already built, and that was a dangerous path. I had built something actually good. I know, because it worked, and a complex system working like that = it's good.
I also know how it works. I was there for the failures. I fixed them. Seriously. I could explain the architecture. I already invested the required time to make it reliable enough that I was willing to share it with actual participants and not just friends I wrangled into pilot testing.
And again, I am a social scientist.
Unfortunately, it still wasn't producing the evidence I needed. So I'm contemplating redesigning a lot of it. Maybe (read: hopefully) not all of it. Some infrastructure has got to make it to the new study.
And I did actually learn a few things during all that troubleshooting. I am now much savvier about building participant-facing interactions, preserving interaction histories, moving people between systems, capturing intermediate states during a task, and structuring conversational data for analysis. The implementation might be changing or going away, but I still did the work.
I'm telling myself that I don't have to insist that every piece of that work remain visible in my final studies in order to have been worthwhile.
I'm not sure this particular kind of research mess gets talked about enough.
We all share that the study was conducted. We write up our methods and describe our system as though it emerged fully capable. Sometimes we throw in a sentence or two about pilot testing or methodological refinement. We don't generally include the time we had something complicated and functional and cool that we were proud of, and then had to decide if it functioning was enough.
It's not always enough. Sometimes you discover conceptual problems. If all this piloting helps you become better at executing the study, the limitations might actually become more clear. So I have to choose to be a responsible scientist and acknowledge that months of troubleshooting might not lead to just one more fix.
I'll probably just get to build something else.
So I'm choosing to frame this as a helpful "look at messy research before we clean it up for publication" and not as a whole bunch of wasted work. I built something and learned things that helped me get the thing to work, and things that showed me that the thing I made work wasn't the thing I need.
And then you realize this will make a great entry for your research notes, so take that, infrastructure! Even more benefit out of the theoretically useless-to-my-study system!
Sarah, remember: this is allowed to be unfinished. If it were finished, it wouldn’t be a Research Note.
As always, these notes reflect my own intellectual work. AI was used for organization and conceptual scaffolding, both as support of and as an object of iterative inquiry into the creative process and cognition.