mixed effect models

Funnily, mixed effect regression was the first type of regression analysis I learned (I was given a huge complex data set with no prior R experience as an analysis task). I compiled a collection of papers and link and books that I used to self teach. My goal is to provide the links and a description as to why they were useful/why I needed that information so one can follow along and self teach also.

I am also continually updating it as new sources arise.

To begin, the following are MUST READ books or papers or tutorials. They really set the foundation for understanding and building multilevel models and what they capture beyond regular ordinary least squares regression (they aren’t necessarily in reading order, links may need to be updated too).

The following are extra materials that are highly relevant but that I didn’t interact much with. They may (or may not) be useful.

Finally, here are some pages that go over some of the basic questions I had during implementation. I will try to cluster them into overarching topics.

Understanding the analysis

Of course, the first search I did was to understand mixed regression in general. What does it do? Why do I need it? How is it different from other analyses?

Confidence Intervals

After running your regression, how do you get confidence intervals for your betas? Typically you use confint(model) or if you want wald (asymptotic and fast but less precise) confidence intervals, you use confint(model, method=’Wald’). However, here are some links for comparing confidence intervals through other packages or the difference between prediction intervals and confidence intervals.

(Restricted) Maximum Likelihood Estimation

An important aspect to understanding these models is how the parameters are estimated (hint: not using least squares). They use Maximum Likelihood (ML) or Restricted Maximum Likelihood (REML).


I understand the lack of p values in these models, but I come from traditional labs, so I had to learn how to draw p value based inferences from these models. There are many methods for this: likelihood ratio test (lrt) for model comparison, lmerTest for both anova and predictor style inference, bootstrapping, etc.

Logistic Regression

I ended up modeling trial accuracy data, which is a binary outcome variable and thus requires logistic regression models. The implementation wasn’t difficult, but interpreting the results takes practice and care. These links are general tutorials that helped me understand implementation and coefficient interpretation.

Model Building

I keep getting mixed advice about this approach and its varieties. I was taught by a statistician who said stepwise approaches were ok but I read otherwise. For exploratory work this may be ok (as compared to confirmatory), but do what you want. I’ll just post the materials I used to understand these methods.

Model Complexity

When I first started, I wondered how crazy these models can get. Can I just throw every variable in? Are there costs/benefits/limitations to parsimony vs complexity?

Model Fits

Diagnosing whether the model fits well and how to do so is important. This typically involves some form of checking unexplained variance along with examining assumptions.


When the data is not robust enough for the model or the model is too complex, it will not converge. This tends to render your estimates unreliable. So this is an important issues to either fix or look into to see how bad it is.

Variance Components

I’m currently working on projects that are more interested in the variance components than the betas. The variance components tell you how much the means vary across units of your random effects, e.g., if participants is a random effect, how much their intercepts vary. Important to this topic are intraclass correlations (ICC) and variance partitioning coefficients (VPC) and their interrelations.


Related to variance components, the within/between subject variance can give you a sense  about the reliability of your measure. The within subject variance would be the residuals that aren’t captured in the model, the between would be the random effect groupings. Not all links are necessarily mixed model related, but may be useful. Note: this is intimately related to variance components/icc above so those sources will also help.

Power Analysis

The hardest part (for me) about starting a study is determining power, especially when your analyses consist of complex mixed models. I haven’t fully read through all of these links,  but I am aggregating them to read soon.


This is an approach I’m slowly starting to look into, how to make my multilevel models bayesian. Here are some packages that are helpful.


I have started examining how to model longitudinal data.


I have also started examining how to model nonlinear data (e.g., prediction accuracy, psychophysical data, etc).


Turns out analyzing your own data is only half the applications. Sometimes it’s worth it to be able to simulate data sets with specific variance components, or even specific crossed random effects. Here I’ll be adding links that help thinking through data simulation, especially when it gets complicated in terms of correlations between participants and/or stimuli.


This is just stuff I learned through the process that may not be directly related to mixed models.

So these are the links I found most useful, and I will update as I continue forward. And when I have more time I will make the links more descriptive as they are cryptic at the moment.

whiteness in statistics

I’ll set the context so people see where I’m coming from. I’m an undocumented immigrant from Mexico, have lived here in the U.S. for 21 years, and my family learned english as a second language. You can say I understand the immigrant/foreign experience intimately. This morning I scrolled through Facebook and came across a question on a statistics forum I follow Psychological Methods Discussion Group. You can read the transaction here (I scrubbed the Middle Eastern woman’s name but kept the repliers’ names for accountability).


There’s lots to deconstruct here.

Here we have a Middle Eastern woman who asked for help about a vague problem in a group whose title invites these sort of questions. Could her question have been more wordy, sure. Is it often that people who are non native english speakers ask questions that can’t directly lead to an answer? Sure. But that’s not the issue at hand. Daniel Lakens and colleagues (Robin Kok, Jazi Zilber, etc; all white presenting, many northern European) took it upon themselves to be the arbiters of the English language and deride this woman’s approach to finding resources. (Jazi Zilber is particularly troublesome given his self-pronounced experience with non native speakers).

When I read through their disgusting mocking of the lack of english proficiency, I couldn’t believe what I was looking at. Not only is it racist to assume that a person of color is lacking proficiency in language (as insinuated by their comments on her punctuation not being used in any language), but the accusation of a mental disorder for a simple question is gross. Yes, I was very aggressive -but think about the context: I’m a graduate student publicly fighting established researchers on their racism + I grew up in poor neighborhoods where physical violence and pointed call outs was the way to handle these problems + it’s sickening to see individuals disrespected. I follow Daniel Lakens and that group because I think they make great points about statistics in psychology. But here is some more context: this particular group has already been embroiled in the big fight about tone policing, with its members being particularly sharp about their academic critiques of others’ work. Generally, I agree with their main points about being vocal of unsound science, but this thread has fallen into an unacceptable domain. See, groups like this are full of psychologists and statisticians, who are “liberal”, and mainly (from what I see – and I read most posts) white. This means there usually aren’t checks and balances to keep their ignorant racial insensitivity in check. If there are people of color, they are far and few in between and clearly unwelcome unless they abide by white academic’s classist and racist rules of engagement. Not everyone in a non-english-dominant country can afford or has access to english education. And even when they do, it doesn’t mean they can hold it above others. Let’s take a look at the english of the man who cast his stone about english proficiency, here a video of one of his talks.

So to recap, we have a white northern european, with that english proficiency, and his fellow bullies jeering a middle eastern nonnative speaker’s english. I learned english as a child, so I grew up highly proficient, but it infuriates me to imagine my mom or my younger self coming into a room titled “HELPFUL DISCUSSIONS” asking a question only to be mocked for a) coming to ask for help and b) how we asked for it. If they didn’t know what she was asking or if they knew of a better forum, they could have asked in a non condescending manner or pointed her in a better direction. Instead, they took it upon themselves to completely ostracize the woman and subjugate her to shitty unnecessary comments. They clearly lack the humility it takes to provide help to those who most need it, I would hate to see him teach minorities. Uli was right, you can ignore the question or patiently engage. Don’t be an ass and try to masquerade it as benevolence.

For those who asked for evidence of how an instance like this could be racist, please read the following paper of the year in the scientific journal Language by Sharese King that delineates how racism manifested in the judgments of a witness’ testimony in the Trayvon Martin case who spoke in African American Vernacular English.

Here is an excerpt:
Screen Shot 2016-12-29 at 2.07.04 PM.png

I also highly recommend the ending of that paper delineating steps for how to fight linguistic racism (in the courtroom), stay woke. In addition, there is emerging field of raciolinguistics.

Here is an excerpt from the blog post “Why we need raciolinguistics“:

Screen Shot 2016-12-30 at 10.13.19 AM.png

The other aspect of this is something I have wanted to write about for a long time. The tone policing and suppression of minority voices. There’s been plenty of times on these forums that I have seen sexist (a man sarcastically dumbing down his remarks to a female professor as if she didn’t understand, it was clear he was wrong), transphobic (a man repeatedly enforcing gender binaries when transgender individuals were also relevant to the topic), and racist comments that go unchecked (like above). Or if they are checked, they are fought back against. Whether these instances are due to explicit prejudicial mindsets is unknown, but what is clear (to me) is often people don’t understand (or care to understand) how what they are saying is oppressive (which further necessitates for them to shut up and listen to critique).

Here are the replies I received:


Now we have a white northern european telling an undocumented Mexican in the United States what counts as racism. So I guess his motto is he can criticize others on their academic work, but he’s immune to critiques on his smug elitism, racism, and classism online. I knew when I commented I was jumping into their bro-den (they usually stick up for each other there), so I was ready for the backlash. However, what I received was far more unsettling. Daniel Laken’s threat to my academic future for calling out a problem he’s propagating is 100% unacceptable. Personally, I’ve fought so many damn obstacles to get to where I am, I don’t have a narrow minded view of success – if a post like this where I expose racism in my field limits my future jobs, then those are places I wouldn’t want to work at. What is the point of a forum or of all these surveys regarding the diversity within the field if not to bring up these issues? How is a graduate student supposed to address what they see as an injustice from established researchers? When it’s not about the research itself, but rather their toxic character, should a student stay silent? Should targeted established individuals be threatening students into silence? I don’t have the answers, but I’m willing to find out.

As a psychologist, this type of public interaction is an embarrassment to the field that studies human minds and needs to be addressed. Is this the reputation we want? Stuck up elitist white psychologists/statisticians who would rather spend time making fun of a help seeker than being mindful and patient or at the very least just not engaging?

To conclude, can someone with clinical developmental psychology experience help this woman? It may not be fruitful, but at this point she’s owed this much. (Update: she has been helped and apologized to)

Also, and this is actually the main point: fellow scientists, do better.

Joel Martinez

For a look at how whiteness defends itself (disguised as scientific inquiry) when a person of color speaks out, take a look at many of the responses in the following thread:

Makes you wonder about the state of this field. Some loud people on that thread (actual practicing psychology researchers) lack basic knowledge of racism, whiteness as a structure and mentality, and minority experience. They lack self awareness to take a step back and see what they’re defending. They were happy to disregard accountability for their friend. They flaunted complicity by pretending there was no issue, that we minorities just need to toughen up. They felt entitled to belittle/discredit/attempt to mechanically quantify the perceptions of racism of a minority. They transferred blame to me as the aggravator for calling out an injustice with a quickness. They gladly fought over whether it was racism or classism as a theoretical issue or whether I was calling him racist rather than focusing on the real experience that occurred. One of my favorite replies accused sensitive academics like me of inciting right wing populism across the world.

These are the tactics used by whiteness to maintain superiority and gas light minorities into submission. (With regards to charges of sexism, there is definitely something to be said about the lack of inclusivity in my approach and it’s something I learned from this situation and will correct in the future. Still doesn’t negate any of the points I have made here).

I limited my engagement in the debate because I had no idea where to even begin, there was so much ignorance of the critical conversations we have in the U.S. regarding race relations, the large and small forms of racism, and the improvement of minority treatment – no easy place to start. The moderators tried to keep it middle of the road, a “sanitized” look at the situation, but people got nasty defensive. Some responders prioritized their “objective” need for evidence over the fact that as minorities, we perceive microaggressions differently (otherwise their legitimacy wouldn’t be questioned). This ends up becoming a colorblind analysis, which may explain why they can see a bunch of white people make fun of a POC’s English and not register it as racism. This also places their lack of experience with racism as the benchmark and minority experiences needing “evidence” of racism to be taken seriously (ironically, the kind of evidence that one builds through personal experience with racism). What an eye opening insight into the nonprogressive mindset of some fellow psychologists. For social psychologists (the field that discovered implicit prejudice), I’m surprised that how they spoke about racism sounded like they expected it to only look like a confederate flag wearing white guy yelling the n word. I clearly struck a chord, and I have a feeling this won’t be the last discussion about this topic given their lack of understanding. There’s a LONG way to go.

One final point, many people supported my post through likes but generally stayed silent (though big thank you to the ones that did jump in). I received private messages of support for bringing this topic to light and if you look at where the support of most people of color lay, it was in likes for this post. Interestingly, they could easily see the racism that the people mentioned above couldn’t. That’s what mattered to me, that it resonated with them. However, to my fellow researchers of color and allies, you need to find confidence in your voice and fight for each other. Social norms matter, and currently they suck bc they allow situations like this, but we can change that. We shouldn’t let people tell us what constitutes racism when they’ve never been on the receiving end of it and we will not be told that our perceptions of racism are imaginary. We finally have a voice and this is the growing pains of battling for visibility.


social mindfulness

Peer-Reviewed Research + Books

My goal here is to aggregate academic resources on social justice. It will be a one page source for scholarly perspectives on social justice. Keep in mind this page is under construction, when I have more time I will fill it with content. Also, it has a focus on issues that I have had immediate experience with or a need to learn, but it will be expanded to include other topics.

The following are links to pages that have already aggregated important publications in list form.

Below I will list individual papers/books and summarize the take away message. I’ll try to find open links to them.

Racial Discrimination


Forms & Norms


Colorblindness & implicit



Incarceration & cop interactions

Politics, Housing, & Income




Costs & Consequences


Conflict Management

Queer Issues





Social Action


International Considerations


Immigrant Health

Immigrant Crime

Immigrant Education

Field Research

Economic Class







Other Sources

Here I will aggregate different sources like blogs or fiction that I have found insightful on the same issues above. Given that these are more author-driven pieces, doesn’t mean I agree 100% with everything they write, but rather, they are on this list because they provide insights that help conceptualize an issue.

Race/Ethnicity & Discrimination


Forms and Norms




Conflict Management


Queer Issues





International Considerations




Economic Class

Critical Theory


On this page, I will aggregate issues related to statistics proper.  For a developed resource page on mixed regression models, go here. This page will cover facts about other statistics – effect sizes, bayesian analyses, homo/heteroskedasticity, etc. Given the need to run high powered studies to output worthwhile science, I will also be collecting links and papers here regarding: how to run power analyses, the concepts behind them, simulations, and general information about them along with issues of replication and reproducibility of analyses.This page is currently under construction.

Error management

  • Family-wise error rate: Link

T Test

Degrees of Freedom

Testing Variance homo/heterogeneity

Effect sizes



Confidence Intervals

Likelihood statistics

Multivariate methods





  • The following site is a tutorial for how to structure your analysis stream to make it more reproducible. Link



P Hacking/QRPs

Machine Learning

  • Machine learning in neuroscience: Link

Signal Detection Theory



hidden minority

I participated in a campaign for the Princeton Hidden Minority Council aimed at raising awareness for what it means to be a first-generation or underprivileged minority going to Princeton. I think it’s important to be exposed to our perspective and the typically unspoken problems we face here in this privilege bubble. Here is my submission:



You should check out the website

You should also check out the album and the other submissions (link). I think they raise important points regarding family, identity, achievement, struggle.



My studies will require faces of all kinds, so I think I’ll just congregate face databases here.

Here’s a database others are working on with more information and sources.

Here’s another that includes more than just faces.

AgingMind – lifespan faces with a wide array of emotions: Link Paper
FACES – young, middle, old with variety of emotions: Link Paper
CAFE – children, many emotions, many races: Link Paper
KDEF – emotional adult faces: Link
Chicago Face Database – different ethnicities, normed faces: Link
10K Adult faces – adult faces, not really normed: Link
CVL – young adult faces, neutral, many angles: Link
FEI – brazilian faces ages 19-40, variety of angles: Link
Georgia Tech – frontal faces: Link
PICS – variety of angles, colors, ages (old looking though): Link
USCD – variety of angles, 28 faces: Link Link2
Bogazici – Turkish face set
FRL: neutral/smiling, diverse ethnicities + ages Link

Less Useful
Youtube faces: Link

If you want a fuller list: Link

compute & model

Being at Princeton, I’m surrounded by computational modeling, so I will aggregate sources here.

Reading list

Network analysis

Neural network

Reinforcement Learning

Dimensionality Reduction

Machine Learning

  • techniques: Link
  • Easyml: Easily Build And Evaluate Machine Learning Models

Drift Diffusion

  • Drift Diffusion model is a computational model of decision making that decomposes reaction times and responses into cognitive components. The linked package is a specific python package I have used before.
  • Heirarchical Bayesian method: Link
  • hBayesDM: Link

Social Modeling

  • David Rand’s behavioral economics code: Link
  • Social Network Analysis class: Link
  • Bootnet: Link


code, gather, process

Here I will aggregate sources related to programming tutorials, gathering data through different mediums and processing that data. This page is under construction.



  • Udacity Tutorial
    • Excellent tutorial that Jon Cohen used for his cognitive psychology course.



Gather Data

Web Scraping



Process Data


  • If you collect data from Qualtrics but analyze in R, you will find that the format you download the data in from Qualtrics is not R friendly. I found this website that gives some great instructions to make the preprocessing of the data that is pretty simple.


  • I use Rmarkdown in Rstudio to create my analysis scripts in a reproducible way, and in a way that allows easy documentation into pdf or html form.
  • Links on how to use it. Cheat sheet. Documentation.
  • I realized that it is possible to write python code in Rmarkdown! Check the deets here. I have been making tutorials for the lab so this will come in very handy. I’m probably behind the times on this, but I figured I would share.
  • Papaya – to write APA style papers
  • Skimr – get the most out of summary statistics of data frames.
  • Write thesis in R markdown


  • If you gather data on online databases, it’s possible it will come in JSON format, which is very not R friendly. However there are some ways to get around this.
  • R package: jsonlite, Link (I use this method)
  • R package: tidyjson, Link
  • R tutorials on hacking through the json data: LinkLink