Preparing for AI audits and customer questions

Not long ago, questions about AI came mostly from enthusiasts inside the business. Now they come from outside. Large customers add AI sections to their supplier questionnaires. Auditors ask how automated decisions are controlled. Insurers want to know where AI is used. Regulators in several jurisdictions have introduced or proposed rules that require organisations to explain their AI systems.

Customers and regulators increasingly ask how you govern AI. Preparation turns questions into routine. Organisations that have their basic records in order answer in days, with confidence. Those that do not spend weeks chasing information across teams, and sometimes discover uses nobody in leadership knew about. This article sets out the records that make the difference.

What people are asking

The questions vary by audience, but they cluster around the same themes. Knowing them in advance shows what records you need.

• Where do you use AI, and for what purposes?
• What data do those systems use, and is any of it ours or our customers’?
• Who is accountable for each system?
• How do you check that outputs are accurate and fair?
• How do you protect the systems and the data from misuse or attack?
• What human oversight is there, and can decisions be explained and challenged?
• What happens when something goes wrong?

Few of these questions require sophisticated technical answers. They require evidence that the organisation knows what it has, has thought about the risks and has someone responsible.

Keep an inventory

List systems, purposes, data and owners. The inventory is the foundation for everything else. Without it, every question starts with a search. With it, most questions can be answered by reading across a row.

Include AI built into software you buy as well as systems you build. A customer service platform with an AI triage feature, an HR tool that ranks applications and a finance system that flags unusual invoices all belong on the list. So do approved general-purpose assistants and the main uses people make of them.

What each entry should record

1. The system or tool, and the supplier if bought.
2. The purpose: what it does and which decisions or tasks it supports.
3. The data it uses, including any personal or customer data.
4. The risk tier, under your internal model.
5. The business owner, accountable for its use and outcomes.
6. The date it went live and the date of its last review.

A spreadsheet is enough to start. What matters is that it is complete and current, so assign someone to update it whenever a new use is approved or an existing one changes.

Document testing

Record how accuracy, bias and security were checked. Telling a customer that a system was tested carries little weight. Showing them what was tested, how and with what result carries a great deal.

For each system in the medium and high risk tiers, keep a short record of the checks made before launch and since. It does not need to be a long report. A page or two covering the following is usually enough.

• Accuracy: what the system was tested on, how performance was measured and the result.
• Fairness: whether outcomes were compared across relevant groups, and what was found.
• Security: how access is controlled and whether the system was tested against misuse, such as attempts to extract confidential data.
• Limits: known weaknesses and situations where the system should not be relied on.
• Oversight: who reviews outputs and how often, and how users can report problems.

For bought software, ask suppliers for their own documentation and keep it with your records. Many suppliers now publish summaries of how their AI features are tested. Where they do not, note what you asked for and what you received.

Report regularly

A short quarterly report builds a track record. It need not be more than two pages for leadership: what was added to the inventory, what was reviewed, any incidents or complaints and how they were resolved, and any changes in regulation that affect your systems.

Keep the format the same each quarter so changes are easy to see. A one-page summary with a short table of systems by risk tier, followed by notes on anything new or unusual, is enough for most organisations.

The report serves two purposes. Internally, it keeps leadership informed and makes AI governance a normal management topic rather than a crisis subject. Externally, a series of reports is powerful evidence. When a customer or auditor asks how you govern AI, being able to show four consecutive quarterly reviews says more than any policy document.

Prepare standard answers

Most organisations receive the same questions in slightly different words. Draft standard answers to the common questions once, approved by the relevant owners, and keep them alongside the inventory. When a new questionnaire arrives, the team adapts existing answers rather than starting again.

Consider a typical software supplier selling to banks and insurers. It used to spend several days on the AI section of each security questionnaire, pulling answers from engineers and lawyers each time. After building an inventory, testing records and a library of approved answers, the same section takes a few hours, and the answers are more consistent and more accurate.

Where to start

None of this needs a large team. In most mid-sized organisations, one coordinator working a few days a month with the system owners can keep the inventory, testing records and quarterly report current.

Build the inventory first: ask each department head to list the AI tools and features their teams use, and record them with an owner. Then pick the two or three highest-risk systems and write a short testing record for each. Schedule the first quarterly report for the end of next quarter. By the time the next customer questionnaire arrives, you will have most of the answers ready.

Source: Intelliversal analysis.

Frequently asked questions

Do we need to include AI features inside software we buy?

Yes. Customers and regulators care about how AI is used in your business, not who built it. Record bought features in the inventory and keep the supplier's documentation.

How detailed should testing records be?

Proportionate to the risk. A page or two is usually enough for medium risk systems. High risk systems may need fuller documentation, especially where regulation sets specific requirements.

Who should own AI audit preparation?

A named person in risk, compliance or IT usually coordinates it, but each system's business owner remains responsible for the accuracy of its records.

Explore AI governance, risk and compliance