[USA] Was dumb. Now scammer knows my full name, address/email, phone number AND took money out of my account. Now what?

I just got scammed while trying to update my health insurance for my new job.

I thought I called the legit number, but I got a scammers network instead. I spoke with this totally legit sounding person about details to my insurance plan for nearly an hour. I gave the person I spoke with my full name, address to my home AND PO box, my email address, and my phone number. I gave them my debit card information, and authorized a transaction via my phone for over $500.00

Yes. I am a total idiot.

So I saw the amount had been taken out of my bank account, but the promised email never went through. I called my health insurance and they said no update had been made to my account. I immediately then went over to my bank and asked them to cancel my debit card which they did. They also had me call a claims person and I told him what happened. They said they will give me a « credit » and put the money back in my account, but if they can’t prove what happened, I owe that money back.

I’m terrified that these scammers will keep contacting me and find ways to ruin my life because of my stupidity. I’m scared that they now have access to every single bit of personal information. I don’t know what else to say except that the number I called sounded totally legit and I fell for it big time.

What do I do?