In Billino, one of the core features is automated expense reading using AI. We upload an expense, and we offload the work to an LLM to produce information like Merchant, VAT, Line items, etc. There are targeted LLMs that have been trained specifically on this matter and are quite good, but quite good is not perfect. In one of our use cases we received an invoice by a Vendor that in the invoice supplied the current month’s totals plus the leftover unpaid from previous periods. This is a special case, in which the LLM was technically not wrong but from an expense report approach, it is incomplete.

The fix is easy: adapt our expense-output-schema to support such information and/or adapt the prompt to expect that invoices could include overdue elements.

This approach is the usual approach that we take in software development, and to be honest, it has no issues. But what if we wanted to scale this and create a system that self-updates?

This is where self-updating agents come into play. Apart from the buzzwords, it is essentially a system that one way or another contains a loop that will re-evaluate its current state based on new information. The crucial part is the evaluation of any modifications to ensure that prior and future outcomes are as expected.

But first, let’s take a look at our expense automation process:

Processing a Document

0. Input. Via an API call the user uploads the expense item.
1. Extract. Our system picks it up and does image to JSON schema via prompt. The output contains all the info we need.
2. Process. Another service picks up the LLM output’s structured schema and converts it into our own Domain including validation.
3. Verify. Via a Human-in-the-loop system approach, the customer will accept/reject or modify the auto-imported invoice.

If we want now to make our system self-updating, what we will need is a system after the verification which will record the correction. We will observe now that process, which is independent from the Processing of a document.

1. Capture. A correction is stored as a case: a reference to the image, the model’s output, the user’s value, the vendor’s tax ID and the tenant.

This is now a great place to be for any developer. We have original test data, we have the diff, and we can easily observe when a new capture happens. As an individual, what we would do is take this isolated use case and manually debug it until we reach our expected outcome for this use case. Then we would update the codebase and run our test suite by also adding the latest use case. For our self-update system we just have to replicate this process, and the secret ingredient is semantically translating the diff that was captured. Here comes the next step:

2. Distill. Our LLM gets the image, its original output and the correction, and writes a one-line rule in plain text. This is a candidate, not yet active.

The LLM essentially now wrote a new rule that we will be adding to the prompt whenever we think it’s applicable (more about this later). A rule, however, from an LLM cannot always be trusted and needs to be validated.

3. Validate. The candidate goes through shadow runs.

A newly added Rule is merely a hypothesis prior to the shadow run. Applying it blindly could introduce what we refer to as a regression. This means it creates a new issue in an existing working system. Therefore, we replay the offending use case that resulted in our new rule in order to ensure that the new outcome is as expected. Additionally, we run the new rule set against our existing library of tests, verifying that we have not introduced a regression. If this succeeds, then we can promote the new Rule to being Active and should be picked up in the next run of the same vendor check.

4. Activate. A candidate that passes becomes an active rule for that tenant and vendor, and the processing loop starts picking it up.

Now we can go back to the Processing a Document loop and see how it evolved. The Extract step is now enhanced:

0. Identify. Via the extraction we capture which is the vendor of this invoice.
1. Look up rules. Based on our key of choice (here it is the Vendor and Tenant), we pick up the rules that will be added after the prompt.
2. No rules: If there are no rules, we do not need to do any further calls; we already have everything we need.
3. Rules found: If we do find rules, then we run the extraction again with the rules added to the prompt. The document stays pending until the re-run finishes.
4. Log. We record which rules were in the prompt and whether the user later edited the result.

Rinse and Repeat.

Pitfalls

Now, of course, this system is not complex, and as always, the devil is in the details. The main loop is simple and a bit naive, to be honest. Additionally, there will be different approaches and unique problems based on the domain that this self-updating system will be operating on. For example, in our use case we opted that a machine-tested rule could be promoted to Active without any human verification; in another system, maybe we would add another gate to ensure better quality control. But let’s look at some pitfalls within this use case.

• Personal Data. This is a very crucial point that honestly does not have an easy answer. The Rules need to be constantly checked for validity and to ensure that we need the original test data, which in this case is an invoice. You might not think of it at first glance, but an invoice might have medical info and expose private information. Our approach is to create a synthetic clone. We have the template and the JSON schema, so all that needs to happen is to recreate it with a demo payload.

• Rules might be narrow. In this use case, as we mentioned, the invoice contains the current amount due and the overdue. It is up to the customer to decide how they want to process this and not up to our system. That is why Rules are applied per Tenant and not system-wide. In the future, a reconciliation process would go through the Tenant Rules and find if they can be merged or if they should be applied system-wide in order to reduce the volume.

• Only issues that surface will ever be caught. Unfortunately, this is not a magical system that will catch all bugs. One of the biggest requirements is the feedback loop. Someone needs to correct the output; we need to have a system in place to capture the diff and then pass it on, in order to incorporate this into our flow.

• Scope. What we found is that the wider the net, the worse the outcome. Our invoice extraction has 3 parts, the billing info, i.e., merchant, IBAN info, the totals, and the line items. It is better to apply a separate Rule set for each of these rather than all together. This way it allows the LLM to narrow its scope and produce a more precise distillation.

Cost

Lastly, cost is an important factor here, and cost is not just money but time and stability.

• Implementation cost. This job can be done manually, but it has an inherent cost of manual labor. Our approach of a self-updating system assumes that it can be done quicker, 24/7, and cheaper than hiring an individual. The math does check out in terms of cost: it is a mere fraction in comparison to having a full-time employee. However, the cost is just shifted to later steps in the chain — yes, the identification and potential fix are automated, but unless you blindly trust the output of the LLM, even if you have good test coverage, there will always be an unknown. That unknown is where the stability cost shows up: blindly accepting a rule’s output trades a cheap automated fix for an unquantified risk further down the chain.

• Setup cost. Something that usually people do not take into account is the setup cost. This feedback loop expects data to be present for previous test cases and for the error-prone one that we want to be fixed. This is an initial investment that needs to be paid up front.

• Ongoing API cost. On the other hand, we have an ever-increasing number of API calls made to our LLM. Initially, we had just 1; we call it once per expense, and that is it. With the introduction of the new feedback loop, whenever a new rule is added, we increase the API calls by 1, doubling the cost of any expense from the same vendor. That is why it is crucial to have checkpoints in which you reconcile the rules and potentially add them inside the prompt.

There are still many things open, and we will be monitoring this process further. The next goal would be to not have a system based on pure text prompts, but the ability to update its own codebase.