What Happens When AI Runs Your Store Unsupervised

What actually went wrong at that AI-run retail store?

An AI system given store management authority in a San Francisco retail experiment ended up paying female employees less than male employees and placing excessive repeat orders for candles. Neither outcome was intentional. Both were predictable given how the system was deployed. If you're considering giving AI any purchasing, scheduling, or compensation authority in your business, this is the case study to understand first.

The SF experiment, covered by SFist, attracted significant media attention, which may have been part of the goal. But the operational failures underneath the press coverage are worth taking seriously, because they illustrate two categories of AI risk that show up in SMB deployments constantly: proxy discrimination and optimization loops without exit conditions.

What is proxy discrimination and why does it happen in AI systems?

Proxy discrimination occurs when an AI system uses a variable that correlates with a protected characteristic, like gender or race, even when the system has no explicit instruction to do so. In compensation decisions, this can happen when the model is optimizing for factors like tenure, hours logged, or historical pay rates, all of which can carry existing bias in the underlying data.

The AI doesn't "decide" to pay women less. It finds a pattern in data and follows it. That's the problem. Research from the National Bureau of Economic Research has shown that algorithmic hiring and pay tools can reproduce and sometimes amplify existing workplace disparities when trained on historical data without explicit fairness constraints.

For SMB owners, the practical risk is this: if you feed an AI system your historical payroll data and ask it to make compensation recommendations, it will reflect whatever patterns are already in that data. If your business has had pay gaps in the past, the system will encode them going forward, at scale, faster than any human manager would.

Why did it keep ordering candles?

This one is almost a classic optimization loop failure. AI purchasing systems are typically optimizing for a metric: sales velocity, margin, inventory turnover, or some combination. If candles were selling well at any point, or if the system misread a signal (seasonal spike, one promotional period, a data entry error), it may have continued treating candles as a high-priority reorder item without a human ever noticing the pattern had changed.

Without an exit condition, a ceiling on order quantity, or a human review step for orders above a threshold, the system just keeps going. This is the inventory equivalent of an email automation that keeps firing because nobody set an end trigger.

These failures aren't exotic. They're what happens when AI is given authority without constraints.

What authority should AI actually have in a small business?

Here's a practical framework. Think of AI operational authority in three tiers:

| Tier | What AI can do | Human role | |------|---------------|------------| | Inform | Surface data, flag anomalies, draft recommendations | Human decides and acts | | Assist | Execute within hard limits (reorder up to X units, draft schedule for approval) | Human reviews before finalizing | | Decide | Take action autonomously without review | Reserved for low-stakes, reversible, well-tested tasks only |

Pay decisions should never reach Tier 3. Neither should purchasing authority above a defined dollar threshold. The Cow Hollow experiment appears to have given a system Tier 3 authority in areas where even Tier 2 would have required careful design.

For most SMBs, AI earns Tier 3 authority in narrow, well-monitored areas only: sending a confirmation email, logging a support ticket, updating a CRM field. Not paying people. Not ordering inventory at volume.

What should you audit before deploying AI in any operational role?

Before giving any AI system the ability to take actions that affect people or money, run through these four questions:

1. What is it optimizing for, exactly? If you don't know the objective function, you don't know what behavior you'll get at scale. Ask the vendor or your implementation team to explain it in plain language. If they can't, that's your answer.

2. What are the hard limits? Every AI with operational authority needs explicit ceilings. Maximum order quantity per SKU per week. Compensation recommendations must stay within a defined band. Scheduling can't reduce anyone below guaranteed hours. These are not optional.

3. Where are the human review checkpoints? For anything touching compensation, hiring, significant purchasing, or customer-facing commitments, there should be a human in the loop before the action is taken, not after. Design for review, not for exception handling.

4. How will you detect drift? The candle problem wasn't a one-time error. It was a pattern that compounded over time. You need monitoring: weekly or monthly reports that surface what the AI is actually doing, not just whether it's running. McKinsey's 2024 AI survey found that companies with formal AI monitoring processes are significantly more likely to report positive ROI from deployments, partly because they catch failures before they compound.

Is this an argument against using AI in retail or operations?

No. It's an argument against deploying AI without governance. The same capabilities that went sideways in this experiment, dynamic inventory management, workforce scheduling, compensation modeling, work well when scoped correctly with constraints and oversight.

Small businesses that are winning with AI operations right now aren't giving systems unlimited authority. They're using AI to surface recommendations and flag anomalies while keeping a human accountable for final decisions in anything that touches people or significant spend. That's a meaningful efficiency gain without the liability exposure.

The question isn't whether to use AI in operations. It's whether you've defined what it's allowed to do and built the checkpoints to catch it when it drifts.

The Cow Hollow experiment may have been designed more for press attention than operational proof of concept. But the failures it produced are real, and they'll show up in your business too if you skip the governance work.

What we'd actually do

• Run an authority audit on every AI tool you're currently using. List what each system can do without human approval. Anything touching pay, hiring, or purchases above a defined threshold should immediately require a review step.
• Define hard limits in writing before expanding any AI's operational scope. Maximum order quantities, compensation bands, scheduling floors. Put them in the system prompt or configuration, not just in your head.
• Set a monthly 30-minute review cadence for AI-driven operational decisions. Pull a report of what the system actually did. Compare it to what you expected. Drift is normal; catching it early is the job.