Finn Forecasting AI Agent Launch

Building an AI Finance Teammate

What would you do if your CFO asked you to build a finance teammate from scratch using AI? Where would you start? You might think about the main types of work people do in a company’s finance department. Let’s boil them down to the top three, according to Microsoft CFO Amy Hood.

1. Looking Forward
2. Looking Backward
3. Making the Next Best Decision

In terms of which one is the most important, my vote would be for looking forward. Making decisions about the future is the essence of finance. Looking backward is for accountants. What makes finance fun is using data to make decisions about an uncertain future.

In order to make decisions about the future, you have to make a guess or prediction about what the future might be. This is the practice of forecasting. In finance our forecasts are based on numbers, which are in the form of a time series (numbers over time). Being able to forecast these time series effectively allows the CFO and the rest of the company to make the best decisions. Therefore the most important job you can do in finance is to make high quality time series forecasts.

Creating these forecasts are hard though. They take time, and can often be inaccurate. Thankfully time series forecasting is a perfect problem for AI and machine learning to solve. So when starting to build an AI teammate, the first thing you want it to be good at is the problem of forecasting.

Challenges in ML Forecast Adoption

Historically the practice of using machine learning in financial forecasting has been hard. Both in creating accurate forecasts and in getting finance humans to use them to drive business decision making.

The first challenge relies in knowing how to use the right grain of data and machine learning techniques to get an accurate forecast that can match or exceed the accuracy of existing manual forecasts created by humans. This takes a unique combination of data and machine learning knowledge, as well as the domain expertise of the business you are trying to forecast. Finance people have the domain knowledge, but lack the required data engineering and machine learning knowledge. In the past this means that expensive engineering teams would have to be created to put together these complex machine learning forecasts. These engineering teams would have to do a lot of trial and error around testing what data inputs and machine learning techniques produce the highest quality forecasts. This process can be very tedious. Which slows down how fast these forecasts can be created and how fast new forecast projects can be onboarded to a machine learning approach.

Once these machine learning outputs are created, the other major problem lies in getting finance humans to use the outputs created by a machine. With their manual forecasts created in excel, they can trace how the forecast was created cell by cell. Fully understanding the assumptions and thought process that went into the forecast. With machine learning, this excel model is now abstracted away behind the curtain of a machine learning model. A black box that only a talented data scientist can understand. This creates a problem of trust between a finance human and the machine learning forecast. Because they cannot understand it as easily as their manual excel process, they are less likely to use it for their decision making. Even if the machine learning forecast is proven to be more accurate, they are still hesitant.

These two problems have slowed down the adoption of machine learning forecasts in finance over the last decade. Good advancements have been made in democratizing machine learning tools to help reduce the need for engineers, but the accuracy and trust problems still persist. What we need is to not democratize machine learning to finance humans but to instead delegate the complex work of creating these forecasts into a digital finance employee. An AI teammate who at its core knows how to use AI and machine learning to help finance humans get more work done. This teammate is an AI agent, one who can work 24/7 at scale in the cloud alongside you. Allowing finance humans to do more than they ever thought possible.

That’s the purpose of recent work we’ve been doing in Microsoft finance to build the first version of our forecasting AI agent. It builds upon our existing machine learning forecast framework called Finn, and adds a new AI agent on top of the framework to take forecasting to a whole new level. Let’s explore what the initial version of the agent can do.

Iterate Forecast

The first problem we wanted the agent to tackle was to automate the process of experimentation in trying to find the right combination of input data and machine learning techniques to create the most accurate forecast. This process is easily the biggest bottleneck in starting any new forecast project with machine learning.

We call this the iterate forecast skill in the Finn agent. The first thing it does when starting the iteration process is to teach itself about the historical data. It runs various exploratory data analysis steps to learn many things about the historical input data. For example, it will check for missing values and outliers, seasonal patterns, and what variables are most correlated to what we’re trying to forecast.

Once the agent has taught itself about the data, it thinks through what combination of data and machine learning techniques it should use in the first iteration of the machine learning forecast process. It bases this decision on what it learned during exploratory data analysis, the knowledge it has about using Finn, and custom domain knowledge we gave it based on our experience running thousands of forecast experiments over the years.

After the agent determines what inputs to use, it kicks off a machine learning run inside of Finn. Running through each stage in the Finn forecast process.

1. Data Cleaning: fill in missing values, fix outliers, etc.
2. Feature Engineering: transform the data before training models using lags, rolling window periods, etc.
3. Model Back Testing: train various models on historical data.
4. Best Model Selection: blend different models together and choose which model or combination of models are the most accurate.
5. Final Forecast Output: consolidate the final results into a dataset that contains historical back test and future forecasts.

When the Finn forecast process is finished, the agent will analyze the back testing results and make a decision to either stop or think through another set of inputs to try in a second iteration based on what it learned from the first run to improve accuracy. What determines if the agent keeps running or not is a set of constraints and goals we provide the agent.

• Iteration Constraint: The number of times the agent will iterate on running forecast experiments. Looping through reasoning what inputs to try next, running the Finn forecast process, then analyzing the back test accuracy.
• Accuracy Goal: The mean absolute percent error (what % the forecast is off on average) the back test forecast should be lower than.

The agent will continue to iterate on the forecast until it either reaches the iteration constraint, beats the accuracy goal, or runs out of ideas of what to try next (quits the run). These built in stopping mechanisms ensure the agent doesn’t run forever and gets us closer to our overall accuracy goals within a reasonable time limit.

Update Forecast

The iteration process can take a while to run, based on the size of the data and the constraint/goal you provide it. This makes it hard to use during tight forecast cycles where a finance team may only have a day or two to create a forecast end to end. That’s why we built the update forecast skill in the agent. This allows the agent to take already trained models from previous agent forecast iterations (or updates) and feed the latest historical data through them.

After updated forecasts are created, the agent will analyze the accuracy of this newest forecast with the accuracy obtained during the previous agent forecast iteration. If accuracy is considerably worse for most of the data, if allowed, it will kick off a new forecast iteration process to ensure the agent is creating the most accurate forecast on the data. This is totally optional, and controlled by the user of the agent.

Being able to easily update forecasts based on previously trained models allows us to cut down the total run time by up to 90% when compared to previous Finn version runs. This allows users of Finn to go from creating forecasts in hours to now creating forecasts in minutes. A true order of magnitude improvement.

Explain Forecast

The first two skills in the agent help us solve the first challenge of machine learning adoption around accuracy. What’s missing is being able to easily explain the outputs of the agent to a regular person, without the need of an expert data scientist. This is where the explain forecast skill comes to save the day.

When a user asks a question of the agent, it can think through the problem step by step and come up with an approach to answer it.

1. Planning: The agent first takes the question and thinks through what steps it needs to accomplish to answer the question.
2. Execute Steps: Then it will go through each one of these steps, writing and executing code to get data outputs for final analysis.
3. Finalize Answer: After all steps in the plan are complete, it will synthesize the outputs from each step into a simple answer back to the user.

This type of thinking process allows the agent to answer a various range of complex questions.

• Historical Data
  • Are there any missing data or outliers in the data?
  • What external variables are correlated to revenue?
• Forecast Output Review
  • What was the historical back testing accuracy?
  • Which models performed the best?
• Run Settings and Model Selection
  • What data cleaning strategies were used in the final forecast?
  • What outside variables were used in the final forecast?
• Model Explanation
  • What variables are the most important in the model?
  • Explain how the arima model works to produce the forecast?

Now finance humans can ask the agent any question about the machine learning forecast and get back answers in simple terms that enable them to understand exactly how the forecast was created. This removes the black box complexity and helps build trust. The agent will patiently answer any questions asked and is always available to chat 24/7.

Final Thoughts

With an AI agent, we can now have forecasts that are accurate, fast, and easier to understand. This is the holy trinity of machine learning forecasting in finance. Solving these three areas removes 99% of common adoption roadblocks. This creates an exciting new world where finance humans can closely collaborate with AI teammates to do things never thought possible.

Once we are able to change the finance forecasting game with agents, it makes it easier to tackle the other two components of corporate finance work. If you can use AI to look into the future (forecasting), it makes it a lot easier to look into the past (financial close). Imagine having AI perfectly understand the forecast, which allows it to explain variances to forecast when it comes time to close the books at month end or quarter end. This is the true long term vision of these agents, expanding their capabilities until they turn into well rounded finance teammates.

The future of finance work is very exciting. If you’re interested in the Finn AI agent, check out the Finn open-source documentation to learn more. May the MAPE ever be in your favor.