A systems-level look at a California-based STEM Ph.D.
Positionality: I am writing from the angry but grateful perspective of someone who was lucky enough to finish their Ph.D., recognizing that many others weren’t this fortunate.
Maturity Level of This Essay: Budding [0]
When I was four, I declared my desire to become a scientist. At age eight I played Beyond Good and Evil which kickstarted my desire to create games. From then onwards, I worked towards becoming a game scientist to improve how games are developed. I was lucky enough to be admitted to a Ph.D. program in Computational Media at UC Santa Cruz. After almost five years of playing, thinking, experimenting, and writing, I became a Doctor of Games.
Armed with the experience of the past five years in the academic system, I can confidently say that my corner of the Academia, and my Ph.D. within it, was exploitative, pointless, and noisy–and yet, I would do it all over again.
In this essay, it is not the people in academia I am criticizing. Instead, I am arguing against the organizations, incentive structures, and traditions in which people are trying to survive. Ph.D. students have around five times increased depression risk compared to the general populace, with 1 in 4 Ph.D. students showing clinically significant signs of depression. [1] It doesn’t take an expert to see that there are deeply rooted failings in the academic system.
While reading this critique, it is also important to note that there are hundreds of different academic fields, with their own priorities and traditions. Context matters. I am writing with my perspective gained from a US-based technical Ph.D. Yet, I believe all of academia shares some aspects of the systems I critique.
I hope that this essay will contribute to the conversations we have about the systems we create and will be a helpful data point for those who are just now deciding to step into academia.
Academia is Exploitative
Every first Friday of the month, our Graduate Student Association would host a gathering. The cheap beer was a good reason to go. After the event closed down, we often migrated to the bars in downtown Santa Cruz. I remember loaning my friend a twenty-dollar bill one of those Friday nights so he could join us. He literally had no money left after using a significant amount out of pocket to attend an academıc conference that was required for his work.
In 2016, at UC Santa Cruz, 11% of the graduate students were homeless, and around 40% of them were suffering from food insecurity. [2] I started intermittent fasting to avoid racking up lunch bills as I spent more than half of my income on rent. As a response to this economic pressure, the graduate students went on strike for a living wage, and over 100 of us got fired. Others have told the story of the UCSC wildcat strike, with its financial hopelessness and the militaristic police response in much more detail. [3] What I want to highlight, instead, is how money is moved around and earmarked which creates this economically oppressive situation.
In the US, most Ph.D. students earn their livelihood either through teaching, fellowships, or doing funded research. Only half of the money students earn goes into their pockets. For me, this was around 2000$ monthly, which was lower than the “minimum liveable amount” found by the University of California themselves[4]. The other half is used to pay for the tuition of the Ph.D. student.
This structure is somewhat reasonable at the start of the Ph.D. journey. A first-year Ph.D. student has to take classes. These classes are taught by professors and facilitated by admin that need to be paid–tuition in this context makes sense. However, what doesn’t make any sense is that this structure stays the same even after the Ph.D. student has stopped taking any classes and has effectively become a researcher doing work for their lab and institution.
In the last two years of my Ph.D. I did not take a single class. Yet I was, due to creative waiving and accounting, paying 2000$ per month for tuition. During this time, I had a single weekly one-on-one meeting with my advisor. If my tuition was directly given to him, I would have paid my supervisor 500$ an hour every time we met. Even worse, maybe before COVID, the tuition would have afforded more amenities, but since 2020 I exclusively worked from home, a significant amount of the time using my own laptop and resources,
After several years of work, the Ph.D. student stops being primarily a student that takes classes and needs hands-on guidance. We grow into someone who teaches classes, creates knowledge, and guides others. In my last year of my Ph.D., I was meeting multiple junior Ph.D. students to support them. Yet the system is blind to this change because it is financially convenient. Can you imagine a context where, after half a decade of experience and a direct change in job description your compensation barely grows? It is academia. I wish I could say that the exploitation stopped when you earned your Ph.D. Unfortunately, this is far from the truth, and the system is masterfully crafted to put pressure at every level.
Many who fail to secure a tenure track position continue their academic career as Adjunct Faculty. They have almost no job security and earn very little money [5,6]. These overworked and underpaid folks teach most of the classes in our “world-class institutions.” Often their contracts have to be renewed quarter to quarter, and they have to teach at multiple institutions to make ends meet.
The lucky few who have achieved a tenure track position have seven years to prove their merit by publishing as many papers as possible. Academic institutions have an up or out system: you either get promoted, or you get fired. You can imagine how much pressure this puts on junior faculty, which gets passed down to Ph.D. students, which gets passed down to undergraduates.
Unfortunately, the systematic exploitation doesn’t end there either! Let’s briefly look at the bread and butter of academics–academic papers. Your mandate as an academic is to collect as many citations as possible. The citation count is the one single metric that is primarily optimized for. Apart from citing Goodarts Law [7] I won’t delve deeper into the effects of having citations count as a broken incentive here (as it will take a whole essay). However, I can suggest Anarchistic Fragment 22/23 of James C. Clark[8] if you want to read more on this topic. Instead, I want to briefly trace how labor and money flow through this broken system.
Let’s say, through some means, you as an academic are funded and have been able to write a paper. Great! Now the paper is sent off to a peer-reviewed conference. This means that a group of “peers,” other academics in your field, are asked to take time off from everything else they have going and decide whether this paper is worthy of being accepted to the conference. Are these reviewers compensated? Of course not.
Let us assume the reviewers decided to accept your paper into the conference. Now you have to register for the conference; otherwise, your paper will not be published. If you are lucky, your department will pay for the registration and all the accompanying travel resources; it likely won’t if you are not in a STEM field.
Thankfully now that the paper is published, everyone can read it, right? Of course not. To read this highly selective academic research, you either need to purchase the right to access the document or be a part of a university that has an agreement with the publishers.
So one would expect that you, the academic who has done the research, has reviewed the work, and has paid the publication and conference fees, would get some portion of the journal money, right? Of course not. All the money stays with the publishers.
And yet the free labor doesn’t even stop there! A Ph.D. student has to have their dissertation read by several faculty members to graduate. On the side of the faculty member, this situation means that they commit to reading, understanding, and responding to a document that is usually around 200-300 pages. Of course, no one is compensated for this work, and the lack of compensation often shows in the student’s feedback.
In the academic machine, everyone, Ph.D. students, lecturers, and tenure track faculty, does their best, yet the systems we inhabit lock each individual in a never-ending rat race. What’s worse, this is a rat race only the financially privileged can take place. If you can not sustain your life with the salary of a Ph.D. student for five to seven years then a life of academia is not for you! Do you have dependents, a child, health problems, debt payments, or other financial troubles? Better luck next life.
Academia is Pointless
So, if people stay in this exploitative system, then their work must be endlessly meaningful, right? Whether your work is meaningful depends on how you define your meaning, and I know many who are fulfilled by their work. Unfortunately, as I reached the end of my Ph.D. I realized I was not one of them.
When I was eight, my goal was to “improve games.” Indiscriminate improvement turned out to be rather tricky, so I narrowed it down to focus on making games more inclusive–I simply didn’t want people to feel helpless when they were playing games. The title of my eventual dissertation ended up being “Design Out Helplessness: AI Interventions in Game Inclusivity.” In the end, after five years of effort, I can proudly say I have helped a total of zero people in overcoming their feelings of helplessness. Zero.
I am under no illusion that research results in impact immediately, nor have I any quarrels with long horizon basic research that might or might not materialize. A lot of research endlessly inspiring to me, whether it is basic research trying to uncover the underlying texture of the world, life, and humans, or applied research informing new technologies and guiding policies.
However, my narrow corner of academia, Technical Games Research, has an implicit promise of improving the craft of games. Unfortunately, thanks to the incentive structures we are in, I don’t believe my research will be able to deliver that promise. Let me be more specific. I don’t think the papers I have published exploring videogame inclusivity will make games more inclusive.
The reason is simple: The papers I write about improving games are not read by those who develop these games. More importantly, the papers I write aren’t even for the game developers to read. I write these papers, so people who write about making games read them. Why is this the case? Once again, the incentive structure of academia.
Academia rewards accruing citations and not much more. Even if a game designer pays 30$ to buy the right to read my article, goes through eight pages of dense academic writing, and then decides to improve their game using my work, this has zero impact on me as an academic. Unless the game designer cites my work in a follow-up peer-reviewed paper, my impact on the games industry is invisible. Unfortunately, those who make games do not publish peer-reviewed academic papers that cite other work. Delivering on the implied promise of my field (a game developer utilizing my work) has almost zero benefits for me. So, I don’t put effort into delivering on the implied promise of my field; I focus on writing papers and optimizing for citations.
This misalignment of incentives causes technical games research to slowly drifts away from the actual craft of making games. We write technical games papers for those who write technical games papers; we pat each other on the back and give each other citations. We are never forced to validate the solutions we are providing outside of academia.
Aren’t there papers that manage some change in the industry and game researchers who build games themselves? Of course there are! Yet, due to the structures in place, these are much less than they could have been. From the view of the academic system, the games we develop aren’t recognized; the tools we build aren’t recognized; the game design communities we create aren’t recognized–unless, of course, one manages to publish a paper about these. The system’s single-minded focus on citations ensures that anything that doesn’t lead to papers is underprioritized. Beyond the lack of recognition of all this labor, this misalignment is likely severely reducing the effectiveness of our research. [9,10]
For example, as a part of my research, I built a tool that makes it easier for game developers to build better tutorials. We published a paper out of it! I was incredibly enthusiastic about supporting the tool and getting people to use it. Yet, the verdict was that it wouldn’t be the best use of our time as no explicit research questions were being asked. The tool we developed ended up helping no one. Thankfully the paper got a few citations so that’s nice. These observations won’t generalize to every field, but I offer them nonetheless as an invitation to reflect on the point of research.
Once again, I have to reiterate, everyone I have met in the last five years is doing their damned best, most people bordering on burnout from being overworked. Yet, these brilliant minds aren’t making the most of the incentive systems surrounding, shaping, rewarding, and prioritizing this monumental effort. This magnitude of the problem worsens when one realizes how much uncertainty there is in academia.
Academia is Noisy
I am using the term noise in the statistical sense: “unexplained variability within a data sample.” In colloquial terms, luck. Human decision-making is inherently noisy and filled with bias. [11] This fact becomes especially relevant when the success or failure of your work depends on a rather small number of people: the two or three people reviewing your papers or dissertations.
During the 2014 Neurips, a very selective and highly reputable artificial intelligence conference, an experiment was conducted to better understand the levels of noise in the paper review process. [12] Some of the submitted papers were sent to two separate groups of reviewers. In 25% of the cases, both groups of reviewers agreed that the paper should be accepted, and in another 25% of the cases, both groups of reviewers agreed that the paper should be rejected. However, in the remaining 50%, while one group of reviewers accepted the paper, the other rejected it. This result means that unless your paper is exceptionally good or bad, you flip a coin with every submission.
Last year we submitted a paper to a highly reputable technical games research conference. The first reviewer stated that they did not know enough about the technical aspects of our method, so they decided to give a neutral vote. They gracefully indicated that they liked the writing in our paper. The second reviewer stated that they liked the approach and the results, but they gave a reject vote due to finding the paper’s organization confusing. The last review was incredibly positive, singing the praise of a paper accompanying a strong acceptance vote. Sadly, however, the reviewer had made a mistake and reviewed another paper–the strong praise had nothing to with our work. Our paper got accepted. [13]
This situation was incredibly baffling to me. What was worse, when I wanted to discuss the situation with my larger community, the sentiment was confusion as to why I was making this big of a deal. Didn’t the paper get accepted after all?
This amount of noise would have been acceptable if you had a high number of attempts. The law of large numbers states that one can get to the true underlying mean given a large enough sample size. Unfortunately, however, in research, you get a somewhat limited number of shots. Often Ph.D. students submit one to two first-author papers per year.
Even if we move past the noise contained in the human evaluators, there also is an immense amount of uncertainty in research, almost by definition. If one was able to state which research questions are worthwhile before exploring them, it wouldn’t be called research! By sheer luck disguised as intuition, you could pick a very fruitful research domain or wander around for years on a barren topic. Of course, as one matures as a researcher, one’s intuition sharpens, but the uncertainty never disappears. Realistically, however, starting Ph.D. students don’t yet have the intuition to pick promising research fields–at least, I didn’t. Thankfully the Ph.D. advisor is always there to offer guidance, which leads us to the next source of uncertainty: the compatibility of the student with the advisor.
Does the student prefer joining a pre-existing established project, or does she want to start something on their own? Does the advisor have a strict, organized approach, or does she prefer a more free-flow approach? Do the priorities and values of one align with the other? There are no correct answers to these questions, just productive matches. The answers to these questions affect the success of the research lab drastically. Unfortunately, these questions are often not considered in the application phase and are left to chance.
According to the New York times in the US, the average length of a Ph.D. is eight years. [14] This is as long as the mean marriage length in the US! [15] When one decides to get married, hopefully, they have months, if not years, of experience getting to know each other. Often a Ph.D. student and advisor agree to work together after exchanging only a few emails. Once again, at least in certain US institutions, the admission system leaves a lot of these crucial compatibility questions to luck. For example, I decided to change labs at the end of my first year due to an incompatibility between myself and my advisor. I believe it was one of the three most important decisions I took in my Ph.D., and I probably would have not been able to finish my Ph.D. otherwise.
Most Ph.D. students aren’t aware of how big of a role luck plays in their results, which can result in immense feelings of doubt and imposter syndromes. “My papers aren’t getting accepted. The reason must be because I am not smart.” Furthermore, research is incredibly personal and emotional. For better or worse, often, people pour their identity into the ideas they are defending and presenting, which makes the noisy rejections sting even worse and likely contribute to the increased mental health issues.
And yet, I would do it all over again
I am grateful for my time during a Ph.D. It was incredibly enriching. Most importantly, it gave me time. Time to ask questions that didn’t pan out, time to talk about failure for months on, and time to contribute to a larger ongoing conversation. It got me a well-paying job. I have a cool-sounding title. It worked for me. [16]
I state that I would do it again, because: I didn’t have medical bills that were prohibitive. I didn’t have dependents that I need to take care of. I didn’t have a sense of financial expectation that was put on me by my culture or parents. I had an incredibly supportive supervisor and cohort. I lucked out on finding a reasonable research topic. I lucked out on matching with reviewers with whom my work resonated. I lucked out to be in a situation where I didn’t have to suffer from racism or sexism.
I hope I made it clear that getting a Ph.D. has an incredibly high variance, and unfortunately, I believe an overall negative expected reward. Even though most suffer, some end up extremely fulfilled and successful. Whether this is a risk you want to take is only up to you. I hope that this systems-level look at how academia is structured will help you make a more informed decision. I put all my skill points in luck in character creation, rolled the dice, and scored a critical success. If you roll the dice, I hope luck smiles upon you as well.
More importantly, however, systems can change.
Let’s do better. I had said “Let’s do better.” in the initial version of this document. However, it was to lessen the bridges I was burning and give this whole clusterfuck of a situation a hopeful spin. After more conversations with people I trust, I realized I was being weasely and dishonest. Especially being able to write this from a position of privilege, having been able to receive the title.
I am not trying to do better. Because I got out. I am working in a high-paid job in the games industry, doing research into tools that will reach players. So maybe, let me be braver and update my call to action. Let’s leave academia, and take our skills, passions, and research elsewhere, where it will be recognized and rewarded. Then maybe, the pressure will get academia to up their game.
Batu Aytemiz
Thanks to Sudikchya, Breanna, Oskar, and Grammy’s House for comments!
[0] This is the first entry on my “digital garden”. The posts here aren’t static, but will grow as I have more conversations and thoughts around it. I am planning on three levels of maturity, seedling, budding, and evergreen. I was inspired by Maggie Appleton, read more about their gardens here: https://maggieappleton.com/garden-history
[1] https://www.nature.com/articles/s41598-021-93687-7#Sec2
[2] https://twitter.com/leyabreanna/status/1292897163719000064 / https://www.universityofcalifornia.edu/about-us/information-center/student-basic-needs
[3] https://payusmoreucsc.com/
[4] https://financialaid.ucsc.edu/cost-to-attend/graduate-costs.html
[6] https://www.chronicle.com/article/how-to-fix-the-adjunct-crisis/
[7] Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.”
[8] https://theanarchistlibrary.org/library/james-c-scott-two-cheers-for-anarchism#toc32
[9] https://notes.andymatuschak.org/Insight_through_making
[10] https://www.econlib.org/call-it-sour-grapes/
[11] Noise: A Flaw in Human Judgment, Kahneman, 2021
[12] Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. Cortes and Lawrence, 2021. https://arxiv.org/abs/2109.09774
[13] I have a proposal on how to change this in our field! Moving to guidance over gatekeeping format. I will convert this thread into a post soonTM. https://twitter.com/BatuAytemiz/status/1475510479825027074
[14] https://www.nytimes.com/2007/10/03/education/03education.html
[15] https://legaljobs.io/blog/divorce-rate-in-america/
[16] A repeated critique I got was that this paragraph was weak. I, uh, yeah. Decided to write separately about the nice parts of my journey later.
Garden 