The aim of this study was to explore the association between three previously identified dietary patterns (Western, Prudent, and Mediterranean) and colorectal cancer (CRC) risk by sex and cancer subtype. The Spanish cohort of the European Prospective Investigation into Cancer and Nutrition study provided dietary and epidemiological information from 15,629 men and 25,808 women recruited between 1992 and 1996. Among them, 568 CRC cases and 3289 deaths were identified during a median follow-up of 16.98 years. The associations between adherence to the three dietary patterns and CRC risk (overall, by sex, and by tumour location: proximal and distal colon and rectum) were investigated by fitting multivariate Cox proportional hazards regression models stratified by study centre and age. Possible heterogeneity of the effects by sex and follow-up time (1–10 vs. ≥10 years) was also explored. While no clear effect of the Prudent dietary pattern on CRC risk was found, a suggestive detrimental effect of the Western dietary pattern was observed, especially during the first 10 years of follow-up (HR1SD-increase (95% CI): 1.17 (0.99–1.37)), among females (HR1SD-increase (95% CI): 1.31 (1.06–1.61)), and for rectal cancer (HR1SD-increase (95% CI): 1.38 (1.03–1.84)). In addition, high adherence to the Mediterranean pattern seemed to protect against CRC, especially when restricting the analyses to the first 10 years of follow-up (HR1SD-increase (95% CI): 0.84 (0.73–0.98)), among males (HR1SD-increase (95% CI): 0.80 (0.65–0.98)), and specifically against distal colon cancer (HR1SD-increase (95% CI): 0.81 (0.63–1.03)). In conclusion, low adherence to the Western diet and high adherence to the Mediterranean dietary pattern could prevent CRC, especially distal colon and rectal cancer