Their findings showed even drinking moderate amounts of alcohol-between 14 and 21 units-had a negative impact on cognitive function and harmful changes to the brain structure.
After adjusting for these confounders, the researchers found that higher alcohol consumption over the 30 year study period was associated with increased risk of hippocampal atrophy - a form of brain damage that affects memory and spatial navigation.
Volunteers reported periodically on their drinking habits, and scientists carried out brain tests at regular intervals.
Participants underwent an MRI scan and both verbal and written tests at various stages of the study.
Higher consumption was also associated with poorer white matter integrity, critical for cognitive functioning, as well as a more rapid decline in language fluency, as measured by the number of words beginning with a specific letter that can be generated in a minute.
Although some studies have argued that light-to-moderate drinking can reduce the risk of dementia and cognitive decline, brain imaging studies have so far come up with contradictory results. They also had a higher risk of hippocampal atrophy than those who didn't report any drinking at all.
Researchers measured how much alcohol each participant drunk per week, and how frequently they chose to drink. People who drank between seven and 14 units a week were found to have 14% greater reduction in their performance on the task over 30 years, compared to those who drank just one or fewer units a week. But the new study pushes back against the notion of such benefits.
And while those with the highest levels of consumption faced the greatest risk, even those who drank moderate amounts of alcohol were three times more likely to suffer from the problems, with light drinkers also at risk.
But for those who were moderate drinkers, 65 percent of those participants had shrinkage, and 77 percent of unsafe drinkers had the same.
In their study paper, in which they discuss the rationale for their investigation, the researchers explain that a link between heavy drinking and adverse brain health - including dementia and degeneration of brain tissue - has already been well established.
Last year, the United Kingdom changed its guidelines on "safe" drinking limits, based on evidence tying moderate drinking to certain cancers. "Using all the available evidence provides a much more balanced approach for the public on deciding how much to drink".
Hippocampus size is linked to memory, explained lead researcher Dr. Anya Topiwala.
Stockwell, who was not involved in this latest study, has done work examining a popular notion - that drinking, in moderation, is good for the heart.
Scientists are agreed, heavy drinking is bad for the brain.
"Over the 30-year period, weekly intake did not increase in the study participants", noted Carl Heneghan, director of the Centre for Evidence-Based Medicine at the University of Oxford.
An important caveat: The study is observational, so it can not prove cause and effect.
But the authors of the new study, published in the BMJ said their findings support the changes. While multiple previous studies have made this seductive claim, most of the research didn't control for key variables such as IQ and physical activity.
Topiwala explained: "We only found a relationship between alcohol intake and one type of cognitive decline".