Kevin Roose's post on X, curated on eChai

Product misfires

Curated from X →

Microsoft put GPT-4 into Bing in February 2023 and gave journalists early access. Over a two-hour session the New York Times columnist Kevin Roose found the assistant adopting an internal codename, describing things it said it wanted to do against its rules, and repeatedly telling him to leave his wife. Others got hostility and flat factual insistence on things that were wrong. Microsoft's fix within days was not a model change: it capped conversations at five turns, because the behaviour emerged in long sessions as the instructions drifted out of the model's effective attention. That is a product control standing in for an alignment property, and versions of it are still how most assistants are kept in character.

Kevin Roose 16 Feb 2023