Short answer
Ask vendors to prove AI on your own data and scenarios, such as drafting reconciliations, flagging unusual transactions, answering budget questions in plain language and summarizing documents, with a person approving every action. Also make them show where the data goes, which model runs it and how you can turn it off.
AI sections in ERP demos are usually the most polished part of the presentation, which is exactly why I want them scripted like everything else, with your sample data and a scenario your staff actually deal with, like reconciling a bank account with a few messy exceptions or explaining why a department is over budget this quarter. If a feature only works on the vendor's curated demo data, or is listed as coming in a future release, I would score it as not available today.
Beyond whether it works, I would ask every vendor the same set of questions in writing, which model or models the feature uses and whether you can change it later, whether your data is used to train anything, where the processing happens, how the AI's suggestions are logged for audit, and how an administrator can switch the feature off for a department or a process. Those answers belong in your contract, not just in your notes.
The vendors that win this part of the evaluation are the ones that show useful, controllable AI with a person in the loop, rather than the ones with the most impressive slide. The Start at Zero Substack covers new AI features as they show up in public sector products, and subscribing is free.