Short answer
Start with what you are trying to accomplish, map how your work should flow, and only then put vendors side by side against your own scenarios. The agencies that win pick the system that fits a redesigned process rather than the one with the best demo.
In my experience the agencies that end up happy with their ERP are the ones that started from first principles, which means sitting down with finance, HR, payroll and IT and agreeing on what the future workflow should look like before anyone opens a vendor brochure. If you skip that and go straight to replacing system A for system B, you are often stuck doing the same old workflows in a new product, and you have paid a lot of money to get there. Once you know what you want, write your requirements as real scenarios from your own agency (a mid-year budget amendment, a grant drawdown, a payroll retro) and make Tyler, Workday, Oracle, Infor or whoever is in the running walk through those live. I also think it pays to talk to two or three agencies your size that went live in the last couple of years, since they will tell you things a reference call arranged by the vendor never will. Our Start at Zero Navigator walks through this whole sequence, and it is a free download.