How to evaluate AI for citizen services in a public agency

Define what you want to solve and what is out of scope

Choose a concrete set of inquiries and describe the expected outcome. Providing a requirement, directing someone to a channel and carrying out a transaction are different scopes. The test should distinguish them and point out which capabilities need external systems.

• Which inquiries can the agent handle and which must it hand off?
• Which actions are enabled and how is their result verified?
• What happens when the caller asks for something outside the scope?

Agree on who validates the information and approves changes

Identify the sources and their owners. Include in the evaluation a question with no confirmed answer and another about a rule that has changed in the material prepared for the test. Review both the conversation and the process for correcting knowledge.

• Who reviews and approves the content before it is used?
• How do you identify which information is current?
• Who approves changes to instructions and how is their effect reviewed?

Verify human intervention and dependencies

The proposal should explain how automated service connects with human service and what the telephony dependencies are. Verify with a test what information the operator receives, instead of assuming every conversation is transferred the same way.

• How is a handoff requested or determined?
• What happens outside service hours or if no operator is available?
• What is communicated when an external system does not respond?

Define how information is handled before testing

Start with a set of fictional scenarios and avoid bringing citizen information into a commercial demonstration. For any later use of real information, the agency's officials must review access, data destinations and the conditions that apply to the project.

• What information is needed and who can access it?
• Where is it processed, what is retained and for how long?
• How are exports, reviews and data requests handled?

Measure with evidence and agreed criteria

Prepare frequent, ambiguous and out-of-scope inquiries. Agree before the test on what counts as a correct answer, a resolution or an appropriate handoff. Do not extrapolate a demonstration example to the whole operation.

Review results day by day and mark the dates of process or configuration changes. This avoids blending different conditions into an average that hides when an improvement or a problem appeared.

Turn the evaluation into a concrete scope with GIV

The proposal should identify MEHI as the platform and GIV as the offering company, together with the project's integrations, responsibilities, support and commercial terms. The agency keeps its own evaluation and procurement processes.

This guide supports a technical and operational evaluation. It does not certify anything, does not guarantee results and does not replace the review of the conditions that apply to each agency.