(1) Background: The aim of this study was (i) to analyze problems faced by informal caregivers in three areas of their life: health, work and finances, and family and social relationships, (ii) to investigate the main determinants of these problems, and (iii) to explore differences between men and women. (2) Methods: The study population consisted of people aged ≥18 years living in a family home who were providing unpaid care to a dependent person in the same or another home and who were registered as caregivers with the Primary Health Care District of Granada or the Provincial Council of Gipuzkoa. Several logistic regression models were built to analyze the likelihood of caregivers experiencing health, work-related, or social problems as a result of their caregiving responsibilities. (3) Results: Informal female caregivers were more likely to experience problems attributed to caregiving than their male counterparts, particularly in the areas of health and work. Additional factors associated with an increased likelihood of problems were low perceived social support, performance of ungratifying tasks, and fewer years as a caregiver. (4) Conclusions: Informal caregivers in Spain face significant problems as a result of their caregiving duties, and the impact on men and women is different. Policies and interventions to mitigate the negative effects of unpaid caregiving should incorporate differential strategies to meet the specific needs of male and female caregivers in different caregiving contexts.