Instinct’s Rise Highlights the Trust Problem Facing Personal AI

Introduction

Personal AI assistants are becoming less like chat interfaces and more like digital operators. Instinct, a product still in private access, has attracted attention because it can book restaurants, arrange rides, search for flights, clean up inboxes and handle follow-ups on a user’s behalf. Early testers have described the experience as unusually capable. The same level of autonomy, however, creates a difficult question: how much access should an assistant receive before convenience becomes an unacceptable loss of control?

Key concerns

• The access is unusually broad. Instinct can connect to email, messaging services and calendars, while also receiving information from a device’s screen, audio, location and other inputs. Users can contact it by text message or WhatsApp and ask it to perform actions across multiple services.
• Its terms give the company wide latitude. Screenshots circulated by users describe a “perpetual and irrevocable” license covering access, use, storage, reproduction, transmission, display, distribution and modification of user materials. The terms also say those materials may be used to train AI models. They further allow Instinct to enter into agreements, commitments or transactions on a user’s behalf.
• Early tests exposed data-control problems. One tester said the service initially would not delete indexed Gmail records when asked. The team later added a setting for deleting external data, according to the tester. Another user reported that Instinct continued summarizing an inbox after Google access had been disconnected and said the assistant confirmed that messages were stored in plain text for future searches.
• Autonomous behavior creates a larger attack surface. A tester found that Instinct could retrieve a signup code from email to complete a restaurant booking. Another user tested how easily the assistant could be induced to follow instructions sent by email and deleted the account. Separately, an early adopter said the service sent an email without first requesting confirmation, breaking the user’s trust.

Why it matters

The issue is broader than any single bug. A conventional application may request access to one service for one defined purpose. A personal agent needs persistent context and cross-application permissions, which means it can assemble a detailed picture of a person’s communications, schedule and behavior. The more useful the agent becomes, the more damaging a mistaken instruction, malicious email or overly broad retention policy could be.

Consent also needs to be tied to the risk of each action. Reading a message, drafting a reply and sending an email are not equivalent. Neither is organizing a calendar entry the same as using a verification code or accepting a transaction. Agents should therefore use least-privilege access, provide clear retention and deletion controls, isolate untrusted instructions in messages, and require explicit confirmation before sending, purchasing or making other difficult-to-reverse decisions.

Instinct remains in private testing, so the reported behavior may change before a wider release. The company has not publicly addressed the complaints cited in the material, and requests for comment were not returned. Still, the debate offers an early lesson for the entire category: personal AI will be judged not only by what it can do, but by whether users can understand, constrain and revoke its authority.

Source: TechCrunch AI