How to fight the irs

Audits Soar Among the Rich; Taking the Tax Man to Court

Tom Herman

Updated Feb. 18, 2006 12:01 a.m. ET

It's the taxpayer's nightmare: Duking it out with the tax man.

The Internal Revenue Service has stepped up its attempts to rake in more overdue taxes, and it isn't being shy about auditing more people and clamping down more frequently with tactics such as levies and even asset seizures when it thinks someone needs to cough up more money.

IRS officials have particularly zeroed in on individuals making $100,000 and above. Last year, more than 219,000 people with incomes in that group were hit with an audit, up about 32% from the previous year.

Driving the crackdown are not only concerns about federal budget deficits but also evidence of cheating and other forms of tax-dodging. For example, the IRS says millions of people who should be filing returns, don't. A few days ago, officials estimated that the government is losing about $290 billion a year to cheating and other forms of noncompliance.

The IRS is especially concerned about self-employed workers. But it has also launched assaults on what officials call "abusive tax shelters," which are transactions the IRS insists have no business purpose other than avoiding taxes. They are also attacking other techniques used by the super-rich, including family limited partnerships.

Some tax-shelter battles already have produced huge inflows of revenue to the Treasury. For example, the IRS said that last year, more than 1,200 people accepted its offer to settle battles over a shelter known as "Son of Boss." The IRS also said it has collected nearly $4 billion from that group.

With the IRS intensifying its crackdown, growing numbers of Americans could soon be facing a tough decision: Whether to fight back, or simply cry uncle and cut a check.

While the notion of fighting the IRS strikes dread into the hearts of citizens everywhere, the fact is that many people actually stand to benefit by doing battle with Goliath. There's an array of options, ranging from a simple appeals process to taking the tax man to court.

For people in particularly dire financial straits who can't pay what they owe, there is a special compromise program where you can bargain for a better deal. On average, people accepted into that program end up paying pennies on the dollar. (The bad news: It's exceptionally tough to be accepted into the program, since it requires proving extreme hardship.)


And even if a taxpayer does something clearly wrong, such as missing a key deadline, the IRS may be willing to remove penalties if the taxpayer had a good reason, such as a death in the family or major illness, says Linda Martin, a director at Deloitte Tax LLP in Washington and a former IRS official.

If the IRS is coming after you, in many cases giving up can be a smart move, particularly if the stakes are low. Tax law is complex, and many taxpayers don't have the time or stomach for what can be a long and expensive struggle. Here's how to decide when picking a fight might be worth the trouble.

The first step in deciding whether to fight an IRS challenge is to do a quick cost-benefit analysis. Among the key factors: How much money is at stake, how good your records are, how confident you are of winning, how complex the issue is -- and, if you need to hire a tax expert, how much that will cost.

Some people go to court even when the stakes are low. For instance, Jerome J. Norris of Potomac, Md. went to Tax Court to contest the IRS's charge that he owed a measly 48 cents in income tax -- plus a late-payment penalty and interest totaling $440. Mr. Norris called the claim "a colossal blunder."

The case went to Tax Court Chief Judge Joel Gerber, who in his opinion late last year said that Mr. Norris had met his burden of showing the disputed amounts were "in error" and that the IRS hadn't met its burden of showing it was appropriate to impose the penalties in dispute. An IRS spokesman declined to comment.

Other issues, however, can be much more complex. An example would be a case in which a couple that had filed jointly gets divorced, and the IRS comes after one of them for taxes that person feels are owed by the other.

One piece of advice: Never sign a joint return if you suspect your spouse is breaking the law. Instead, consider filing separately. Because once you sign a return, you're generally on the hook if something goes wrong later, unless you can show you were an "innocent spouse" -- which can include proving that you didn't know, and had no reason to know, about any tax shenanigans. These cases are tough to


Whatever the case, if you get an IRS letter challenging something on your return and asking for information, "the worst thing you can do is to ignore it" and hope the issue will disappear, says James A. Dougherty, a former IRS official and now a director at Deloitte. Even though the IRS's computer systems are notoriously antiquated, officials are likely to catch up with you, and interest and penalty payments can pile up.

Once you respond, if the agency won't budge or seems to ignore your point, consider asking to speak to the auditor's supervisor, Mr. Dougherty says. If that doesn't work, consider other paths, such as:

APPEALS: This is an IRS unit where issues may be resolved by correspondence, phone or in person. Taxpayers who want a face-to-face meeting can get one, the IRS says. There's a special procedure for "small" cases, those in which the total amount for any tax period isn't more than $25,000. You may represent yourself or use an attorney, certified public accountant or an "enrolled agent" (someone specifically approved to practice before the IRS). For details, see IRS Publications 5 and 556 at .

The IRS appeals office receives about 100,000 cases a year, and roughly 80% are resolved, says Bruce Friedland, an IRS spokesman.

IRS TAXPAYER ADVOCATE SERVICE: This is a good place to consider going if you've already tried regular IRS channels and feel that you are stuck in the bureaucracy. It can be especially helpful if you are facing a financial hardship because of some action the IRS has taken or may be about to take. Even though it may seem counterintuitive to seek relief from the IRS itself, the "TAS" unit has drawn widespread praise from tax professionals for its independence and willingness to help taxpayers.

In the year ended Sept. 30, taxpayer-advocate offices around the nation received roughly 200,000 cases.

To contact the office, call 877-777-4778, or go to There is at least one advocate office in each state.

TAKING IT TO THE COURTS: Most people choose the U.S. Tax Court ( ) since you don't have to pay the contested tax up front in order to have your case heard.

The court hears a wide variety of cases involving such areas as income taxes, estate taxes and gift taxes. But the court can't hear cases involving certain other issues.

Some people prefer going to federal district court or the U.S. Court of Federal Claims ( ) because they think their chances are better there. But these courts generally will hear tax cases only after you've already paid the tax and filed a refund claim with the IRS. IRS Publication 5 has more details.

If you do go to court, remember that your case and, potentially, sensitive financial details become a matter of public record.

WHEN YOU CAN'T PAY: If you're facing major financial woes, you have a few choices. You can ask the IRS to let you pay on an installment plan. Or, if you're in such dire financial straits that you can't pay everything you owe, consider asking the agency to compromise.

Getting the agency to agree to a compromise is exceptionally difficult and may involve a lengthy struggle. The IRS received about 106,000 offers in 2004 and accepted about 20%. Asking for a compromise should be considered only after all of your other payment options have been exhausted, the IRS says.

Before you build up your hopes for a compromise, keep in mind that this is one of the most heavily criticized IRS programs. Critics say the IRS often takes much too long to reply and rejects too many offers.

IRS officials say that they've improved and that part of the blame lies with the large numbers of blatantly unacceptable offers -- many of them submitted by firms working on behalf of taxpayers -- that slow up the process. They say some of these firms charge excessive fees to people who have no hope of meeting the program's requirements. So do plenty of homework (such as checking with the Better Business Bureau) before forking over money to anyone promising it will be a breeze to get the IRS to buckle.

A FINAL WARNING: Don't even think of challenging the IRS if you plan to argue that paying taxes somehow is voluntary, or that wages aren't really income. Courts have consistently rejected these and other arguments. Some people who tried have been hit with fines of as much as $25,000.

As surprising as it may seem, there is a small but highly vocal number of people who have been trying it, the IRS says. In fact, so many people have tried things like this that the IRS has compiled a list of dumb arguments you shouldn't even bother with. To study up, go to the IRS Web site and type "frivolous" in the search box.


Category: Taxes

Similar articles: