You've got a demo booked with an AI vendor next week. Maybe it's for the maintenance line, maybe it's a chatbot for leasing, maybe it's something for your compliance files. The deck will be polished and the demo will work. That's not the part you need to test. Here are the five questions I'd want you to ask, and the answers I'd expect to give if you asked me.
1. What happens when it doesn't know the answer?
Every vendor can show you the happy path: the tenant asks a simple question, the AI answers it correctly, everyone's impressed. Ask what happens on the question it can't answer. Does it guess, or does it say "I don't have that" and hand off to a person? A system that guesses confidently is worse than no system at all, because your tenant believes the wrong answer until someone catches it. Ask for a live demo of the handoff, not a slide about it.
2. Where does my data actually go?
If the AI touches tenant files, lease terms, or resident phone numbers, you need a straight answer on where that data is stored, who can see it, and whether it's used to train anything beyond your own account. Ask if your data is used to train models for other customers. Ask what happens to it if you cancel. A vendor that can't answer this in one sentence hasn't thought about it enough to be handling your tenants' information.
3. Can I see it get something wrong?
Don't just ask for a demo of success. Ask the vendor to show you a case where the AI got something wrong, and what happened next. If they can't produce one, they either haven't tested it enough or won't show you. Every system that handles real calls or real documents gets something wrong eventually. The question that matters is whether a person catches it before a resident or an auditor does.
4. Who checks this before it goes to a resident or a file?
Ask exactly which outputs go out unreviewed and which ones a person reads first. A maintenance call that gets logged as a work order automatically is a different risk than a lease answer handed straight to a tenant with no review. Get specific: does a person read every overnight transcript? Every flagged file? Or does the system run unsupervised once it's live? If the answer is "it just runs," that's not a small detail, that's the whole risk profile of the product.
5. What does a 30 day test actually cost me if it doesn't work out?
Ask what you're committing to before you've seen it work on your own data. A vendor confident in the product will offer a bounded pilot on one job, one property, or one call type, with a clear exit if it doesn't pan out. If the only option on the table is a full rollout with a long contract, that tells you something about how sure they are it'll hold up.
Where a person stays in charge
None of these five questions are about whether the AI is impressive. They're about who's accountable when it's wrong, who sees the data, and how easy it is to walk away. A vendor worth working with answers all five without flinching, shows you a real failure case, and tells you exactly which outputs a person reviews before they reach a tenant, an inspector, or an auditor. If a vendor can't point to a specific person checking specific outputs, that's the answer to question four right there.
Free 30 minute workflow review
Walk me through how this works on your team today. I'll tell you honestly whether AI helps, and what a small test on one job would look like.
Book a timeHow to test this on one job in 30 days
Pick one job, one property, one call type, or one file type. Write down what a person reviews before anything goes out, and for how long. Run it for four weeks with a named person checking the output daily. At the end, look at what it got right, what it got wrong, and whether the reviewer's job got easier or harder. That's a cheap way to find out if a vendor's answers to these five questions hold up once you're actually using it.
FAQ
Should I trust a vendor who says their AI never gets anything wrong?
No. Every AI system that handles real calls, documents or conversations makes mistakes sometimes. A vendor who claims otherwise either hasn't tested it on enough real cases or isn't being straight with you. Ask to see a real failure and how it was caught.
What's the biggest red flag in an AI vendor pitch?
No clear answer on who reviews the output before it reaches a resident, inspector or auditor. If every output goes out unreviewed and the vendor treats that as a feature, that's the risk you're taking on, not a convenience.
How long should a property management AI pilot run before I commit further?
About four weeks on one job or one property is enough to see real patterns: what the AI gets right, what it gets wrong, and whether your team's workload actually dropped. Anything shorter doesn't give you enough real cases to judge it on.
Is it reasonable to ask a vendor for a reference who had the AI fail at something?
Yes. A vendor who can only point you to success stories hasn't shown you the part that matters most: how they catch and fix mistakes. Ask directly, and ask what changed afterward.