An AI hallucination is a false or unsupported answer that an artificial intelligence tool presents as fact. In a loan file, it can become an invented income figure or a lender rule that does not exist. Check each material claim against its original source before you use the output.
What an AI Hallucination is
An AI hallucination occurs when a tool generates information that its evidence does not support. That can include an invented transaction, a fabricated policy exception or a citation to a document that does not exist.
A language model predicts likely pieces of text from the patterns it has learned and the context it receives. This process can produce a convincing sentence without establishing that the sentence is true. The US National Institute of Standards and Technology describes this risk as confabulation in its Generative AI Profile.
A detailed explanation can contain the same error as the short answer it explains. A citation also needs checking: the source must exist and actually support the claim. Asking the same tool whether it is sure does not replace opening the source.
A misread digit and an invented figure have different causes. Both need correction before the information reaches a loan assessment. The label matters less to the client than whether the broker catches the error.
Where Hallucinations Show Up in Broking Work
Hallucinations cause harm when generated text changes the client’s recorded circumstances or the basis for a lending decision. Every example in the following table is hypothetical.
| Broking task | Hypothetical AI error | What the broker checks |
|---|---|---|
| Bank statement summary | The summary adds a salary credit that the statement does not contain | Locate the claimed transaction in the original statement |
| Payslip summary | The summary describes a one-off allowance as guaranteed base salary | Compare the pay categories with the original payslip and supporting evidence |
| Income and expense figures | The tool adds overtime to income and drops an expense because it assumes the expense has ended | Trace each included and excluded amount to evidence, then recalculate |
| Lender policy answer | The answer invents an exception allowing an application to proceed without a required document | Read the lender’s actual policy and confirm any exception through an official channel |
| Client email | A draft says the lender has approved the loan when the file only shows an application was submitted | Compare the wording with the lender’s actual application status |
| AI meeting notes | The summary says the client wants interest-only repayments after the client said they do not want them | Replay that passage and confirm the client’s preference |
The error can spread beyond its first appearance. A false income figure copied into the CRM can later appear in a lender submission and a client email. Correct every affected record, including calculations that depended on the original figure.
Meeting summaries need the same source check as document summaries. Speaker labels can put a broker’s suggestion into the client’s mouth. Review any statement that records an objective, an agreement or a decision.
Check AI Output Before It Reaches a Loan File
Check AI output by tracing each material statement to evidence and independently recalculating figures used in the assessment. Keep the original documents available while reviewing. Work in the brokerage’s approved system and use only information the brokerage permits that system to process.
Use this checklist before approving the output:
- Match the output to the correct client and source documents. Confirm that the document set is complete and covers the required periods.
- Trace every income, expense and balance figure to its source page or transaction. Check the pay period, gross or net basis and account ownership.
- Reconcile the totals independently. Identify duplicate entries and distinguish transfers between accounts from income or spending.
- Open every lender-policy source. Confirm the lender and applicable product, then read the conditions around the quoted passage. Obtain confirmation for any exception the published policy does not establish.
- Read client-facing text against the file. Remove unsupported claims about eligibility, approval, rates or what the client agreed to.
- Record the correction and approval in the file note. Keep the source locations and the reviewer’s name with the final version.
A hypothetical calculation shows why the pay period needs its own check. A payslip states gross pay of $2,800 twice monthly. That is 24 payments and $67,200 annualised, before considering any other income assessment requirements. Treating the same amount as fortnightly gives $72,800 across 26 payments, overstating the annualised figure by $5,600.
The arithmetic can be correct while the input assumption is wrong. A second AI calculation using the same wrong pay frequency will repeat the mistake. Start with the payslip’s stated period and use an independent calculation.
Make the file note specific enough for the licensee or an auditor to follow. Record which output you reviewed, the source documents and pages, the error and the correction. Identify any remaining question and who must resolve it. Include the reviewer, review date and approved version.
For the hypothetical pay-frequency example, a review note can say, “Compared the extracted gross pay with the payslip on page 1. Corrected the frequency from fortnightly to twice monthly. Recalculated annualised gross pay as $67,200. Income eligibility remains subject to the lender’s assessment rules.”
The final check is whether another reviewer can reproduce each material figure and find support for each policy statement. Keep an unresolved claim out of the approved assessment until the source or responsible person resolves it. ASIC’s RG 273 record-keeping guidance explains why records must show the inquiries and reasoning behind the recommendation.
Reduce Hallucinations With Better Inputs
Give the AI tool the relevant source and a narrow extraction task so its output is easier to check. Separate extraction from assessment. Asking for a whole recommendation invites more unsupported assumptions than asking for the fields shown on one approved document.
For a hypothetical payslip task, use an instruction such as, “Extract the gross pay and pay frequency exactly as shown. Give the page number for each field. Mark any field you cannot locate as missing. Do not infer a value or assess lender eligibility.”
Ask for short supporting excerpts where they help identify the field. Then open the cited page yourself. A page reference can be wrong, and a real excerpt can be applied to the wrong product or pay category.
A fixed-format output makes missing information visible. For example, separate fields for amount, pay frequency and source page let the reviewer compare like with like. Free-text summaries can combine an extracted amount with an unstated assumption in the same sentence.
The fixed format does not prove that the values are correct. A tool can populate the wrong field or invent a plausible value. The broker still checks the source, calculations and meaning before making an assessment.
Use clear copies with readable text and supply the pages needed for the task. When a field is unreadable, obtain a better source or enter the verified value through the brokerage’s approved process. Avoid asking the tool to guess from surrounding figures.
The Office of the Australian Information Commissioner calls for human oversight and verification of AI outputs, particularly where they inform decisions about a person. Keep a human approval point before generated information enters the assessment or leaves the brokerage.
Who is Responsible When AI Gets It Wrong
The broker and licensee retain their applicable obligations when AI assists with the work. Using a tool does not transfer responsibility for a recommendation or remove the need to verify the client’s financial situation.
ASIC’s responsible lending guidance describes inquiries into the consumer’s requirements and circumstances, verification and the assessment of suitability. The best interests duty also requires the broker to put the consumer’s interests first when providing covered credit assistance. An AI summary cannot establish that those steps occurred.
ASIC’s report Beware the gap: Governance arrangements in the face of AI innovation explains that existing obligations apply to AI use. It also states that licensees remain responsible for outsourced functions. A vendor’s accuracy claim does not replace the brokerage’s review controls.
If an error reaches a client or lender, act through the brokerage’s incident process:
- Stop further use of the affected output and identify where it has been copied.
- Preserve the original output and source documents, together with the prompt or processing record available to you.
- Notify the licensee or responsible compliance person promptly. Explain the error and the decisions or communications it affected.
- Correct the records and contact affected recipients through the approved process. State what changed and whether earlier advice or an application needs reassessment.
- Assess client harm and the need for complaint handling or remediation. Have the licensee assess any reporting obligation and record its decision.
An inaccurate AI output does not automatically make every incident reportable to ASIC. The licensee must assess the facts against the reportable situations regime. Escalate the error promptly so that assessment can happen, then confirm that the corrected information reaches every affected client record and recipient.
