Four states have now legalized marijuana for recreational use and 23 states plus Washington, DC have given the green light for medical use. Despite what seems to be a growing acceptance of marijuana in the U.S., the drug is not without risks, especially for the brain.

Now there is more evidence for the idea that regular marijuana use can affect a person’s long-term memory — even after a person has stopped using it.

Those who smoked pot heavily as teens performed worse on the memory test — about 20% worse — and they had differently shaped hippocampi compared to people who hadn’t smoked pot.

Memory problems have always been among the more serious suspected side effects of marijuana, and science has corroborated this connection. Some studies have suggested that heavy users have measurable changes in the area of the brain that governs learning and memory, the hippocampus. The current study is the first to show both effects together — memory problems after years of use and related changes to the hippocampus.

The participants in the study were in their early 20s. Those in the study group had used marijuana daily for two to three years as teens, but were drug-free for two years at the time of the study. Those serving as controls had never smoked pot regularly. Two other groups consisted of patients with schizophrenia, who either had smoked marijuana heavily in the past or had never used it. (There is a known connection between marijuana smoking and schizophrenia.)

Study participants listened to short stories and then had to remember details about them 30 minutes later. They were rated on how many details they were able to recall correctly, which in effect measured their long-term memory. Their brains were also scanned with MRI.

Participants who smoked pot heavily as teens performed worse on the memory test — about 20% worse — and they had differently shaped hippocampi compared to people who hadn’t smoked pot. The more heavily the participants had used the drug as teens, the more pronounced the difference in the shape of their hippocampi.

For people with schizophrenia who had smoked pot heavily, their memory performance was over 25% worse than in schizophrenics who hadn’t smoked pot.

It’s not totally clear which way the relationship works: It could be, for example, that people who had differently shaped hippocampi to begin with were predisposed to pot-smoking. But it’s more likely, the researchers suggest, that marijuana use changes the hippocampus over time.

“It is possible that the abnormal brain structures reveal a pre-existing vulnerability to marijuana abuse,” said lead author Matthew Smith in a statement.

“But evidence that the longer the participants were abusing marijuana, the greater the differences in hippocampus shape suggests marijuana may be the cause.” More research will hopefully reveal how long these effects may last.

Though marijuana is often touted as a “safe” drug, research has consistently shown that it’s not without risks. The poor memory that comes with its use can no longer be called urban legend.

It’s important to keep in mind that synthetic marijuana is not a safer alternative: In fact, it’s much more dangerous.

If you or your kids smoke pot, even recreationally, you might want to reconsider your habit, or talk to your kids about the risks. Though we don’t know everything about how marijuana affects the brain, we do know that it poses risks to one of the most important brain functions we have — our memories.

The study was carried out at Northwestern University’s Feinberg School of Medicine and is published in the journal, Hippocampus.