A Philadelphia woman was jailed in Florida after her ex-boyfriend allegedly used AI-generated texts to frame her, highlighting the growing threat of deepfakes in courts
AI-Generated Deepfakes Impact Courts
Courts are now facing a growing threat: AI-generated deepfakes.
Melissa Sims said her ex-boyfriend created fake AI-generated texts that put her behind bars.
“It was horrific,” she said.
Sims said she spent two days in a Florida jail.
“It’s like you see in the movies ‘Orange is the New Black’,” she said. “I got put into like basically a general population.”
Her story made headlines in Florida.
Sims and her boyfriend had recently moved from Delaware County to Florida.
She said her nightmare began in November 2024 after she called the police during an argument with her boyfriend, who she said allegedly ransacked her home.
“Next thing I know, I’m looking at him and he’s slapping himself in the face,” she said.
She said he also allegedly scratched himself. When police arrived, they arrested her for battery.
As part of her bond, the judge ordered her to stay away from her boyfriend and not speak to him.
Several months later, she said her boyfriend created an AI-generated text that called him names and made disparaging comments.
“I end up getting arrested for violating my bond,” she said. “No one verified the evidence.”
Judge Herbert Dixon says Sims’s ordeal is becoming more common.
“Several years ago, it started out with just fake audio recordings,” he said. “And now it’s gotten to a point of fake video and fake images being produced.”
Dixon is a senior judge on the Superior Court of the District of Columbia. He also serves on the Council on Criminal Justice with former Philadelphia Police Commissioner Charles Ramsey. Its mission is to advance policy in the criminal justice system.
“One of the things we’re trying to do is to develop a framework for the responsible use of artificial intelligence,” he said.
“Do you think prosecutors and police need to do a bit more due diligence in this age of AI before they bring charges, or is that better left for the courtroom?” asked investigative reporter Chad Pradelli.
“Some due diligence has to be involved,” he replied.
Drexel professor Rob D’Ovidio teaches AI forensics.
“This is scary to say, but we’re no longer going to be able to trust what we see in front of us,” he said.
He said AI-generated video, texts, and other evidence can be difficult to spot. AI is simply getting too good.
“The challenge is the detection tools are not keeping up with those capabilities,” he said.
As an example, he created an AI-generated photo to show us. He input it into three different well-known AI detection software programs. All three returned different results that ranged from 1% to 62% probability of the photo being synthetic or AI-generated.
“The standard nowadays is we trust unless proven otherwise, right? I think we have to flip the script and distrust until we verify,” he said.
As for Sims, her story has a happy ending.
After eight months of legal wrangling by her attorney, prosecutors dropped the bond violation charge against her. Last month, she went to trial on the battery charge and was acquitted.
Sims shared her story to advocate for a new law to create AI evidence standards and penalties.
“If this can happen to me, it can happen to anyone,” she added.
In July, Pennsylvania Governor Josh Shapiro signed a new digital forgery law that makes it a felony to create AI deepfakes that injure, exploit, or scam in the state.
Action News reached out to Sims’s ex-boyfriend but has not heard back.
Summary of Key Facts
| Fact | Details |
|---|---|
| Incident Date | November 2024 |
| Location | Florida |
| Duration in Jail | 2 days |
| Charges Against Sims | Battery, Bond Violation |
| Outcome | Acquitted of battery, bond violation charge dropped |
| New Law Signed | July 2023, Digital Forgery Law |
