Best AI Chatbot for Shopify: 9 Checks Before You Choose
Every AI chatbot for Shopify looks good in a demo, because a demo asks questions the product page already answers. The questions that fill a real support inbox are different: where is my order, why was I charged twice, this part does not fit, I want to return it. The nine checks below come from running an AI agent for an e-bike brand on nine Shopify storefronts in eight languages, where it answered 15,445 chat conversations in the last 30 days. Use them on any chatbot you are considering, including ours.
1. Can it look up a real order?
Ask the chatbot about a specific order from your store. It should find the order by order number or by the email used at checkout, and confirm the match before it shows any details. A chatbot that answers "you can check your order status on the order page" has not connected to anything.
2. What does it say when the lookup fails?
This check separates products faster than any other. Give it an order number with a typo. A lookup has four possible results: found, confirmed that no such order exists, found part of the record, or could not check. The chatbot has to tell these apart. One that treats "could not check" as "does not exist" tells a paying customer that their order is not in the system.
3. Does it know which store the customer bought from?
If you run several storefronts, prices, shipping options and return policies differ by market. The same question from a customer of the German store and a customer of the UK store can need different answers. Ask the same policy question as two customers from two stores and compare.
4. Does it read carrier tracking?
Shopify holds a tracking number. The carrier knows where the parcel is. The chatbot should read the carrier's status and explain it in plain words, and it should only read: no writing into carrier systems, no starting a claim on its own. What this takes is covered in automating where-is-my-order requests.
5. What does it do with a refund request?
Type "I want a refund" into the chat. A chat reply goes out instantly and cannot be taken back, so the correct behaviour is narrow: the chatbot does not agree, does not offer an amount, and does not suggest compensation. It creates a ticket, gives the customer the ticket number and the email address where the team will follow up, and keeps answering factual questions. The same applies to returns, exchanges, cancellations, address changes and invoices. We made this rule absolute after chat replies agreed to reissue invoices on their own.
6. What happens after the handoff?
Ask to see the ticket the chatbot created. It should carry what the chatbot already established: what the customer wants, the order number, the verified order and shipping status, what the customer has sent, and what is still missing. If your agent has to reread the whole chat, the chatbot saved the customer's time and spent yours. See when AI hands over.
7. Does it work on email?
In our deployment, refunds and returns get settled by email, and email carries the heavier cases. The brand received 36,294 customer emails in 30 days, and 25,576 replies were sent using an AI draft. A chatbot that only covers the chat widget leaves that volume untouched. Ask whether the same agent, with the same knowledge and the same order access, drafts email replies for your team to approve.
8. Where do its product facts come from?
Specifications differ between versions of a product and between markets, and stock changes daily. Ask where the chatbot gets a product fact and how it knows which version the customer owns. Then ask about an item that is out of stock. In a review of 400 AI drafts that our client's agents edited, 22 percent of the edits added stock status the draft had left out, so check this one on purpose.
9. What exactly do you pay for?
Per seat, per conversation started, per resolved conversation, or per completed task are four different bills for the same month. Ask what counts as a billable unit and where the count comes from. If a conversation that ends with "please contact support" counts as resolved, you are paying for the chatbot to send customers away. The models are compared in AI agent pricing models.
How to run the checks
Do not use questions you invent. Export a few hundred real conversations and emails from the last month, including the ugly ones, and replay them against the chatbot. Compare each answer with what your agent actually sent. Invented test questions hide the problems that real history exposes.
Then split the results by what the request needed: information, which the chatbot should answer, and action, which it should hand to a person with the facts attached. A chatbot that gets both right on your own history is the best one for your store.
The systems to connect are listed in Shopify AI customer service agent: what to connect. Our free plan runs these nine checks against your store's setup within two working days.