Every nonprofit team using AI tools right now is running the same experiment, whether they’ve named it or not: how much do we trust what the tools tell us?
For some tasks, the answer is easy. If you ask an AI tool to draft a newsletter, a grant narrative, or a social post, you already know how to verify the output. You read it. You check the facts. You click every link to make sure it actually goes somewhere. You act as the editor, because you are the editor, and the AI is a drafting assistant, not a source of truth.
But a lot of what AI tools do now isn’t drafting. It’s analysis. It’s “here’s what your data shows,” or “here’s the trend,” or “here’s what changed and why.” Those outputs are harder to fact-check the way you’d fact-check a paragraph, because there’s no red pen for a number. You can’t proofread a conclusion the way you proofread a sentence.
When AI is reporting and doing analysis tasks, you need a real way to decide whether to believe it. Here’s the three-part process you can use.
1. Spot-check the number
Before you trust any figure, ask the most basic question there is: is the number right?
Checking every data point that you just fed into an AI tool would take so much time there would be little point in using the AI. Instead, spot check a couple of data points. Pick one metric and confirm it against the original source. If your AI-assisted dashboard says your email open rate was 34%, go into your email platform directly and check. If it says web traffic from a campaign was up 20%, go into your analytics tool and look at the raw report yourself. Verify the number it tells you clicked “contact us” with the number in your CRM.
Analyzing donations or budgets? Donation and budget figures deserve the same scrutiny, and then some. A mistaken email open rate is an internal correction. A mistaken donation total that reaches a board member or a donor is a trust problem, so these numbers are worth checking every time, not just spot-checking. Better be sure the numbers match before you share them with anyone outside your AI tool.
This step isn’t about distrust. It’s about building a habit. Data pipelines have quiet failure modes: a tracking code that stopped firing, a report that’s counting a different field than you think it’s counting, a tool that changed its definition of a metric without telling anyone. Spot-checking a few numbers every time you run the report helps you catch these failures. Catch them early, and you avoid relying on them for decisions.
If a new intern handed you a report, would you send it to your boss without checking anything in it? The fact that an AI tool will confidently hand you that report doesn’t mean the data is what it should be.
Caveat: this step catches bad data, not bad reasoning. The numbers can all be correct and the story built on top of them can still be wrong. You, the human editor, still have to do your work.
2. Follow the metric
A single number is a snapshot. A trend is a story, and stories are easier to sanity-check.
Whenever something changes in how you’re tracking or doing the work, like a new tagging system, a cleaned-up process, or a new campaign structure, watch what happens next. Does the metric move the way you expected, over time?
If you predicted a change would increase engagement and it did, that’s a good sign the AI’s read on your data lines up with reality. If the metric moves in an unexpected direction, or doesn’t move at all, that’s worth a second look before you build more decisions on top of it. That second look doesn’t have to be complicated: pause before acting on it, and go find out why the trend broke before doing anything else with that data.
This step turns your AI-assisted insights into something you can actually test, rather than something you either accept or reject on faith. Ask your AI tool to flag outliers for you, and look for trends. Then track those trends over time and see if they hold up.
3. Cross-check the sources
The strongest form of trust doesn’t come from double-checking the same tool. It comes from asking a second, independent source the same question.
If one tool says your website traffic from a specific source grew, check whether a different tool that measures a related thing agrees. Two systems that weren’t built to talk to each other, telling you the same thing, is a much stronger signal than one system telling you something twice. Depending on your project, you may be able to find external validating numbers.
This is the same instinct behind getting a second opinion from a colleague, or checking a citation against the original document. Independent agreement is what turns a plausible-sounding number into something you can actually stand behind. Think critically about what the report is telling you and where you can find corroboration. This step takes the most time of the three, so save it for the decisions that matter most, not every routine report – and once you have initially found corroboration, you can think about considering the report routine.
Right number, right trend, confirmed by a second source
Put together, the test is simple: the right number, moving in the right trend, confirmed by a second source. When a piece of AI-assisted analysis clears all three, you can trust it in that report and act on it with confidence.
This framework tells you the data is solid, not that the conclusion drawn from it is the right one. A correct number can still be used to tell the wrong story. These three checks earn your trust in the inputs and outputs. The interpretation is still only yours to make.
This shouldn’t be a one-time audit. It’s a rhythm. Build it into whatever cadence already governs the reporting when AI is assisting, whether that’s monthly, quarterly, or tied to specific projects, so it becomes a human habit.
Verify first. Trust follows, and only for as long as the verification holds up.
Want more practical guidance on using AI thoughtfully at your nonprofit? Visit our AI for Nonprofits hub for more resources, webinars, and tools.
Is Your IT Partner Meeting the Moment?
As your organization navigates challenges, you deserve a support team that is as committed to your mission as you are.
Would you like to see how our human-centered approach can stabilize your IT environment? If you have questions about how to align your technology with your mission, we are here to help.
Community IT has been serving nonprofits exclusively for 25 years. We offer Managed IT support services designed for organizations that want a partner, not just a provider. For a fixed monthly fee, we provide the proactive planning and ongoing IT strategy you need to ensure your technology is always an asset and opportunity, not a liability.
We think your IT partner should be able to explain everything clearly, without talking down to you or using unnecessary lingo. If you’re ready to gain peace of mind and clarity about your IT roadmap in our challenging environment, let’s talk.
As advocates for using technology to work smarter, we’re practicing what we recommend. This article was drafted with the assistance of AI, but the content was reviewed, edited, and finalized by a human editor to ensure accuracy and relevance.
Photo by Javier Haro on Unsplash