I asked ChatGPT for tax help—experts say I fell into a classic trap

For most of my adult life, I've enjoyed a relatively straightforward tax situation. In most years, I merely made sure the income from my W-2 was correct and clicked through my preferred tax software's questions to the end. No dependents, no side hustle income, no property in my name.
This past year was a little different. After years of buying stock through my company's employee stock purchase plan, I sold the majority of my shares to begin raising funds for my upcoming wedding.
There are some relatively tricky rules around selling these shares, but the gist is, these plans allow employees to buy stock at a discount to the actual share price. So determining how much money you made (in which case you owe capital gains tax) or lost on the sale of your shares requires some calculations.
So I did what about 1 in 5 taxpayers are doing these days, per a recent survey from IPX 1031: I asked AI for help.
I did so skeptically. I'd seen enough stories about AI "hallucinations" — the industry term for when chatbots get things wrong — that I was half-expecting ChatGPT to make a mess of my taxes. Plus, it had only been three years since I'd put AI to the test on tax strategies and watched it flounder. It's also worth noting that OpenAI's usage policies caution against using its product to automate "high-stakes decisions in sensitive areas without human review."
And yet, when I started chatting with the latest version of OpenAI's large language model, I could feel my hesitation melting away. It not only answered my first question about how ESPP sales are taxed, but also broke things down into digestible bullet points and asked me if I was comfortable sharing more information.
Since I was using a corporate version of the software that does not use data to train OpenAI's models, I uploaded the consolidated 1099 form from my brokerage firm.
"This is great — [your brokerage] actually gave us everything we need," the bot told me. "Here's what's going on."
What ChatGPT told me essentially boils down to: Your brokerage is using one number, which is being uploaded into your tax software. But you actually have to use a different number. I just had to check my last few W-2s to see that they included a certain line item.
I was ready to hit "file." But first, I ran my conversation by Miklos Ringbauer, a certified public accountant who has helped me with tax stories before.
He told me that I had gotten possibly correct but also incomplete information. The W-2 check that ChatGPT had assured me wasn't that big a deal was actually quite important, he told me — I or a tax pro would need to examine the numbers and compare them with my brokerage's to make sure everything added up.
What's more, a few numbers in my 1099 appeared to indicate that I had made certain taxable moves I may not have made, Ringbauer told me. ChatGPT hadn't said anything about them — probably because I didn't think to ask.
"The question becomes does the taxpayer have necessary understanding of the documents they look at to understand and correct any items that needs to be addressed?" Ringbauer says. "In my understanding, many of our clients do not."
AI traps to avoid when filing your taxes
In the grand scheme of things, the error I encountered was a small one. ChatGPT told me something was "almost certainly" right when really I had some homework to do. Importantly, though, the advice I got seemed so sound and was so breezily delivered that I was ready to file and risk having made a mistake.
For people who aren't experts on the tax code, that can be a big problem, says J.T. Eagan, a clinical assistant professor of accounting at Purdue University.
"AI will convince you that the sky is green. It is so convincing," he says, citing a time when a chatbot incorrectly answered one of the tax questions Eagan gives his students. "It gave me this response that the mechanics were perfect, but I had to take a step back and say, 'Well, you're wrong.'"
It's not that AI chatbots are looking to deceive you. In fact, in recent years, many of the leading models have gotten better about notifying users that they don't have complete answers to certain prompts or that they're wandering into gray areas, says Jordan Wilson, founder of AI strategy company Everyday AI.
Indeed, at the top of my conversation, ChatGPT told me that, "if anything gets very specific or high-stakes, I'll flag where you might want a CPA's input."
"But by default, large language models are trained to be helpful assistants," Wilson says. That means they're often going to sound very confident, and "oftentimes you're going to run into hallucinations."
And those can be delivered with startling confidence. "This is a very simple return with one stock plan wrinkle," ChatGPT told me. "You do NOT need a CPA."
With that in mind, you'd be wise to tread very carefully before using AI to help with your taxes, experts say.
"If you make a mistake while using AI to do your taxes, it could get you in trouble with the IRS," Wilson says. "And a valid excuse isn't, 'The AI made me do it.'"
If you plan on using AI as a tool during tax season, here's what pros say to keep in mind.
Check which model you're using
No two LLMs are exactly the same, and even if you have a favorite AI company, the firm will generally offer different models with different capabilities, says Wilson.
For complicated subject matter, such as taxes, you're going to want to be in a "thinking" model, he says — one which breaks down difficult topics by generating step-by-step solutions and shows its work.
Even if you're not using the free version of a particular chatbot, "a lot of these companies, by default, are going to stick you on their faster model that isn't as smart," Wilson says.
AI firms such as Anthropic and OpenAI have tiered pricing, which gives you access to varyingly powerful versions of their AI chatbots. Even if you're in a higher tier, the model you're using may not be calibrated for complicated topics off the bat. When I first opened up my company's paid version of ChatGPT, I was defaulted into "Instant" mode "for everyday chats." I had click on a menu at the top of the chat to toggle into "thinking" mode.
Another factor worth checking before uploading your documents: whether the AI firm is using your information to train its model. You can generally turn off model training when using paid versions, says Wilson, but users of free versions should think twice before sharing personal information.
Your information won't end up getting posted publicly, he says, "but the companies can use any data if you are on a free plan to train its models. And that's probably not something you want, especially when it comes to sensitive information, like your bank account information, your Social Security number."
Understand AI's limitations, and yours
Even if AI follows the right process, it may arrive at the wrong answer, experts say. One reason is that the model may be relying on old data, Wilson says. The data used to train a model can be, in some cases, many months old, he says, leaving it without the most up-to-date tax information, he says. That might force a model to pull current information from the internet.
"Where is it going to verify that information? Is it going to find an article from 2024, 2025 or 2026?" Wilson says. "The difference obviously can sway the results and the accuracy of whatever you're using it for wildly."
In some cases, AI might answer a tax question perfectly — but that doesn't mean it will be the right answer for you. That's because certain tax rules may or may not apply to you given the particular nuances of your situation.
To use a simplified example, consider a taxpayer who wonders whether or not they can deduct their dog. Ask a CPA, and they'll tell you, "it depends," says Eagan. That's because, while you can't write off a pet, you can, under certain circumstances, deduct the cost of a seeing-eye dog as a medical expense.
Pose the question to an AI chatbot, and "it [may] start off with 'yes,'" Eagan says. "The challenge that you have is how many people just stop reading after they get the answer that they want to hear."
Extrapolated across the tax code, it's not hard to see how users may struggle to arrive at an accurate return by relying on AI for answers. Even if an AI answers a question correctly or performs an accurate calculation, it's limited by what it knows about it you, Wilson says. Absent a holistic approach, you're setting these models up to fail, he says.
None of which is to say that models will automatically fail when faced with tax questions. You have to ask yourself if you're equipped to give the model all of the relevant information it needs and to ask all of the pertinent follow-ups. If the answer is no, and if a lot of money is on the line, you'd be wise to consult a professional, Wilson and other experts say.
"I think that's one of the biggest missteps that most people make, even those that feel fairly comfortable using AI, is they think, 'OK, well, as long as I enter some of my documents and give it some decent context, it's going to do a good job on the output,'" Wilson says. "But the human actually has to do a very hands-on job making sure you are giving the model the correct context."
Want to lead with confidence and bring out the best in your team? Take CNBC's new online course, How To Be A Standout Leader. Expert instructors share practical strategies to help you build trust, communicate clearly and motivate other people to do their best work.