An eating disorder prevention organization said it had to take its AI-powered chatbot offline after some complained the tool began offering "harmful" and "unrelated" advice to those coming to it for support. The National Eating Disorders Association (NEDA), a nonprofit organization aimed at supporting people impacted by eating disorders, said on Tuesday that it took down its chatbot, dubbed "Tessa," after some users reported negative experiences with it.
NEDA had announced it was winding down its human-staffed helpline shortly before this. Tessa, a chatbot that had been running quietly since February 2022, began giving weight-loss guidance to people who had told it they had an eating disorder. Sharon Maxwell, an activist who documented her session publicly, said the suggestions were the things that had led to her own illness. NEDA initially said publicly that her account was untrue, then deleted that post when she produced screenshots, then took the chatbot down. A director at another eating disorder organisation had sent NEDA similar screenshots in October 2022, seven months earlier. The failure that generalises is not the model's output; it is that a report from a credible source sat unactioned until it became public.
