MyCity (Ville de New York)
public-service RAG conversational agent
In March 2024, the city of New York's official MyCity chatbot advised businesses to follow illegal practices, such as taking a cut of tips or refusing Section 8 housing vouchers; the administration kept it online as a pilot.
Objective
Help New York City entrepreneurs navigate municipal procedures and regulations through an official conversational assistant, to relieve pressure on city services.
The deployment
Announced by Mayor Eric Adams in October 2023, the MyCity portal chatbot was meant to guide businesses through municipal rules. In March 2024, an investigation by The Markup showed that it delivered advice that broke the law: it stated that a landlord could refuse Section 8 housing vouchers, that an employer could take a cut of its workers' tips, or that a restaurant could decline to accept cash. Asked the same question about housing vouchers by ten testers, the bot answered ten times that the landlord was not required to accept them, even though source-of-income discrimination is illegal in New York.
Results Proof C
Investigative reporting by The Markup, with reproduced tests and direct quotes, picked up by THE CITY and other established outlets, naming the city of New York and its administration. No published court ruling.
How it works
Inferred typical approachThe internal detail is not public. Here is a proven approach that leads to the same result, to adapt to your stack.
The stack in detail
- plateforme Microsoft Azure AI Microsoft's AI platform that hosts the MyCity chatbot's LLM, according to the city of New York's own communications.
- llm LLM Azure (modele exact non publie) Text-generation model provided through Azure AI; the city did not publish the exact model or the details of the document-retrieval setup.
- outil Portail MyCity The city of New York's official business-services portal, the channel where the chatbot was exposed.
Post-mortem
GraveyardWhat happened sourced
Mayor Eric Adams announced the MyCity chatbot in October 2023 as an aid to businesses. On March 29, 2024, The Markup published an investigation showing that it gave illegal advice: refusing Section 8 vouchers, taking a cut of tips, eviction without process, and no rent cap. The bot was also inconsistent, answering differently across sessions. On April 2, The Markup found it still online, with the administration having reframed it as a pilot with warnings rather than pulling it.
Reason for failure sourced
A public-service conversational assistant that delivered generated regulatory answers with no guarantee of legal compliance, no citation of the applicable rule, and no adequate oversight. In a domain where a wrong answer exposes the user to an illegal practice, the system spoke with confidence about things that were false.
Cost sourced
Reputational and legal cost: an official channel steering businesses toward illegal practices exposes the city to liability. No quantified financial cost published.
Warning signs inferred
Inferred: opening an official regulatory-advice channel with no systematic testing on legally high-stakes questions, no citation of the legal source, and no mechanism to decline when the answer is not certain. The inconsistency across sessions already signaled that the system was guessing rather than retrieving the rule.
Lessons in hindsight inferred
Inferred: in a domain where a wrong answer carries legal consequences, the assistant must cite the applicable rule, decline when it is not sure, and be tested against a battery of trick questions validated by lawyers. A public-service chatbot cannot rely on a confident tone alone; it needs a compliance chain. And when such a system goes off the rails, keeping it online as a pilot shifts the risk onto the user.
Inferred: the public-service conversational-assistant pattern remains valid, provided it is grounded in cited legal sources, bounded to the questions it can handle, and able to say when it does not know. The failure condemns an unbounded generative deployment in a legal domain, not the idea of helping users find their way.
How your customers perceive this type of use
Sourced studiesLes consommateurs n'acceptent pas les chatbots par defaut : 64% prefereraient que les entreprises n'utilisent pas d'IA dans leur service client (Gartner, 2024) et pres d'un utilisateur sur cinq du service client par IA n'en retire aucun benefice (Qualtrics, 2025). L'acceptation se construit sur trois conditions mesurees par Salesforce : savoir qu'on parle a une IA, pouvoir escalader vers un humain, comprendre la logique de l'agent.
Acceptance conditions
- Etre informe qu'on parle a une IA et non a un humain (pres de 75% le demandent, Salesforce 2024)
- Un chemin d'escalade clair vers un agent humain (45% plus enclins a utiliser l'agent IA, Salesforce 2024)
- Une logique de l'agent clairement expliquee (44% plus enclins, Salesforce 2024)
Red lines
- Rendre l'humain injoignable : c'est la premiere inquietude des consommateurs sur l'IA dans le service client (Gartner 2024) et 50% craignent que l'IA les coupe du contact humain (Qualtrics 2025)
- Remplacer le service client par l'IA sans alternative : 53% envisageraient de partir chez un concurrent (Gartner 2024)
Sources: Salesforce 2024 · Gartner 2024 · Qualtrics 2025
How to replicate
Inference, not sourcedData prerequisites
- up-to-date, structured regulatory corpus
- battery of trick questions validated by lawyers
Org prerequisites
- legal validation of the scope
- human oversight and a reporting loop
Possible stack
- RAG bounded to official texts with mandatory citation
- declining to answer outside scope
- periodic legal review
First step: Restrict the bot to questions whose answer is grounded in a cited text, and require it to decline rather than guess; have lawyers validate the test battery before launch.
The same trap, elsewhere
Chegg
verticalized AI assistant built as a wrapper on a third-party LLM
Meta
AI chatbots styled after celebrities
Sources
- S1 NYC's AI Chatbot Tells Businesses to Break the Law Established press archive pending
- S2 Malfunctioning NYC AI Chatbot Still Active Despite Widespread Evidence It's Encouraging Illegal Behavior Established press archive pending
An error, newer info, a source?
This page lives on its accuracy. If a figure has moved, if the deployment has changed, or if you have a higher-quality source, tell us. Every sourced correction is verified before publication.