The software company that created FT Mercati Go to ftmercati.com

Case study

Requirements and acceptance testing with AI

An AI assistant that knows the whole project interviews the customer: it gathers requirements, explains the version under test, records the observations.

In production Custom software: from requirements gathering to acceptance testing of the versions Updated on

  • 16knowledge files in the project, in addition to the system prompt, archived as they are closed
  • 180questions in sixteen blocks in the longest interview, on the most complex topic of the project
  • 5categories for each observation: bug, unexpected behaviour, change request, question, positive feedback
  • 1main question per message, with the answers rephrased and confirmed in the customer's own words
  • 0promises: fixes, timescales, costs and decisions stay with people

In brief

For a custom application, created to replace a spreadsheet the customer had used for over ten years, we built an AI assistant that has read the whole project, from the initial letter to the open questions. Through a personal link the customer talks to it whenever they like: it gathers requirements, explains the version under test, records and classifies observations, asks targeted questions. Everything is saved and read by the developers.

The starting point

A private customer asked us for a custom application to replace a spreadsheet they had been using for over ten years, with layers of formulas and rules that lived in their head. Gathering the requirements through meetings and emails would have taken weeks, and something would have been lost between one phone call and the next. And the acceptance testing of each version would have produced scattered observations, hard to put in order and hard to find again.

What we built

A web application with a personal session link. Behind it, an artificial intelligence (AI) assistant whose system prompt is built from the project materials: the initial letter, the answers to the questionnaire, the specification, the description of the version under test, the calculation models, the open questions. The replies are streamed, the conversation stays saved and can be resumed from the same link, whenever the customer likes. For the developers there is a restricted area with the list of conversations and the detail of every turn, tokens included.

The three phases

  1. Requirements gathering. In the first sessions the assistant interviewed the customer, one question at a time, on the point the preliminary specification left open, the most delicate one in the way they work; out of those conversations came the missing chapter, which went into the final specification, later signed.
  2. Acceptance testing of the versions. At each release the assistant explains what the current version does, using only the material that describes it, and records the observations, classifying them: bug, unexpected behaviour, change request, question, positive feedback. For each one it asks for the missing details and rephrases it for confirmation.
  3. Targeted questions. When a decision is needed from the customer on a parameter, a special case or an extension, the assistant conducts a structured interview: the longest, on the most complex topic of the project, had one hundred and eighty questions in sixteen blocks, with priorities, conditional skips and concrete examples built for the purpose.

How we keep the assistant under control

  1. Understanding only. Never the contract, timescales or costs: those assessments are for people to make, and the assistant says so.
  2. No making things up. It describes only what is in the materials; if it does not know, it says so and flags it to the developers.
  3. One question per message, short messages, examples to be used word for word, no inferring: answers are confirmed in the customer’s own words. These rules came out of the mistakes of earlier sessions, when the assistant tended to rewrite examples and answers in its own words and to infer what the customer had not said.
  4. Everything is read by a person. The conversations are read by the developers; the closed material is archived and the knowledge base updated before the next session.
  5. Visible costs. The token count for every turn is in the restricted area; the context, the same at every turn, stays cached between one message and the next and is not paid for again in full.
  6. An operator’s note, invisible to the customer. From the restricted area the developers can leave the assistant an instruction for the next turn, without the customer seeing it: it is the way to correct course in mid-session.

Declared limits

  • the chat shows plain text, without formatting;
  • without the archiving step carried out by a person, the assistant asks again questions that are already closed;
  • the customer, their data and their sector do not appear on this page: it is a confidential project, and here we describe the method.

What it means for your company

The value is the method: an assistant that has read all the material of a project and interviews people with precise rules, at their own pace, saving every answer. It works for the requirements of a piece of software, for acceptance testing with users, for a customer questionnaire, for onboarding a new colleague. We discuss it in the service on AI assistants that answer on company know-how and in the guide on the AI assistant on company documents. If you have a project in which the information is in someone’s head, let’s talk.

Frequently asked questions

Requirements and testing assistant: questions and answers

How does the assistant know the project?

With every message it has the project materials in front of it: the customer's initial letter, the answers to the questionnaire, the signed specification, the description of the version under test, the calculation models, the open questions. They are numbered text files, maintained by the developers; the closed ones are archived, so that the assistant does not ask again questions that have already been answered.

What can it say to the customer, and what can it not?

It can explain how the current version works, ask, rephrase and record. It cannot make up behaviours of the software, promise fixes, timescales, costs or decisions, or infer an answer in the customer's place: the rules are written in its system prompt and have been refined session after session.

Where do the customer's answers end up?

In a database, conversation by conversation, with the token count for every turn. The developers read them from a restricted area, update the specification or the version, archive the closed material and update the knowledge base before the next session: without this step the assistant would repeat the questions.

Is it only useful for custom software?

No. The same pattern, an assistant that has read all the material and interviews a person with precise rules, works for gathering requirements from a department, for having users test an ERP (the company's management software), for a customer questionnaire or for onboarding a new colleague. The materials and the rules for conducting the interview change, not the tool.

Other case studies

Do you have a process you would like to see working better?

Tell us about it in a first conversation, with no obligation: together we work out whether and where AI brings a measurable benefit, and which step to start with.