The second half of life should be the time that you start to relax a little, when you can enjoy the realization of at least some of your dreams. But coordinating the financial complexities that multiply after age 55 can leave you feeling perplexed and anxious.

How do you know what assets to use first? The sequence of withdrawals can actually amount to millions of lost dollars if you don’t approach it thoughtfully. For example:

  • If you retire early, should you use taxable assets to live on? Deferred comp? Perhaps an annuity you were sold years ago?
  • Do you understand the special rules for retirement accounts that affect cost basis of company stock?
  • Are you confident you won’t pay an early withdrawal penalty if you tap retirement plan assets?
  • Have you thought about where to hold appreciating assets versus income producing assets?
  • Should you let your retirement accounts grow for future generations or would it be better to have growth in taxable accounts and have your heirs get a step up in basis at your death?

Your financial advisor should be your advocate—coordinating among your tax preparer, your estate planner and your investments. They need to know how (or if) you want to leave a legacy.

Let’s explore what you need to consider as you’re plotting out the second half of life!

Accumulation Phase
In your working years, you need to find a balance between spending and saving. Especially in your peak earning years, you need to sock away as much as you can to give yourself more flexibility in life choices later on.

Maxing out on retirement plan contributions is almost always a good idea, but many, if not most, people will also need to save in taxable accounts to accumulate a healthy nest egg that supports their lifetime goals. The money you save in taxable accounts can be accessed at any time and that gives you flexibility. Retirement accounts can’t be tapped until age 59 1/2 (age 55 if you have a company retirement plan and you are separated from service).

Early Retirement
The years between leaving full-time work and age 70 1/2 are some of the most critical from a planning perspective. You haven’t started taking required minimum distributions from retirement accounts, so you have a real opportunity to create tax savings.

Some people are thrilled that they are suddenly in a lower tax bracket after they leave a full-time job and reduce their taxable income. But paying the least amount in tax may actually hurt you!

If you have large amounts stashed in retirement accounts, it may actually be better to start taking carefully calculated distributions to avoid paying much higher taxes after age 70 1/2.

It’s important to think about avoiding the 3.8% Medicare surtax and not paying more in Medicare premiums. But sometimes the “right” answer, from an overall wealth maximization perspective, is to pay these higher taxes. Your advisor will need to consider all relevant factors and use judgement to determine what is best for you considering all your goals.

Another tax planning technique that helps during these early retirement years is “tax bracket” arbitrage. Your advisor can use software to determine how much income to take in earlier years to reduce the overall amount of tax you’ll pay over your lifetime. Obviously many factors must be considered: current and future tax rates, all sources of income and when they must be paid out, income versus estate tax considerations as well as overall investment strategy.

Some studies show the time to be most conservative with your portfolio is right before retirement and for the first few years in retirement. If a bear market hits during these years, it will be harder to recover. If you want to implement this type of strategy, you’ll need to see how you can reposition assets in your retirement accounts first. You won’t have to pay capital gains tax when you reposition in a retirement account.

You also need to consider how different types of assets are taxed. By holding assets in taxable accounts that are subject to capital gains or losses, you will pay a lower rate than assets that are taxed at ordinary income tax rates. Deferred comp plans will pay out and be taxed at ordinary rates. Sales of stocks will be taxed at lower capital gains rates. If you have concentrated positions in company stock or legacy stock, you may want to think about how to gift those assets to other people in lower brackets or to charities or donor advised funds.

Looking closely at your tax return with your advisor can uncover insights about how to better position your assets. For example, if you see a lot of dividend or interest income, you may want to think about stopping reinvestments into income-producing assets. Perhaps a tax-managed mutual fund may make sense. Or that income can be redirected into better diversifying your portfolio globally.

The early retirement years can also be a great time to convert some or all of your traditional IRAs to Roth IRAs. You pay ordinary income tax when you convert, but those Roth IRA assets are not subject to required minimum distributions. You may want to think about leaving them to your heirs. They will also not have to pay tax on withdrawals, although they will have to take required minimum distributions.

If you have a Roth 401(k), you should think about rolling this to a Roth IRA before age 70 1/2. You will have to take required distributions from a Roth 401(k), but not from a Roth IRA.

Social Security can have a million dollar value over a lifetime. So it’s really important to evaluate when the best time is to start collecting benefits. Issues that can affect that decision are the need for cash flow, health factors, spousal considerations and overall tax considerations.

Life insurance has a purpose when your kids are still getting through school or you have a mortgage on your house. But once you’re past those goals, you may want to rethink the need for life insurance. The cost of carrying life insurance often increases dramatically later in life and your advisor can help you evaluate whether you still need as much insurance or if you may be able to convert to a lower cost policy.

After Age 70 1/2
Planning often needs to change once you pass age 70 1/2. Now you will be required to take distributions from most types of retirement plans. For some, doing a qualified retirement distribution to charity may make sense. This technique reduces Adjusted Gross Income (AGI) and that may help avoid additional taxes.

But qualified charitable distributions may not always make sense. If you are trying to maximize charitable donations (perhaps to reduce your taxable estate), you may want Adjusted Gross Income to be higher so that you can take advantage of deducting 60% of AGI in one year.

If you’ve done a retirement spend down analysis in earlier years (and you should!), you may want to run an updated analysis now. You want to make sure you are on track to have enough money to last throughout retirement.

Your analysis can give you a much better sense of “how much is enough?” Of course you want to make sure you can take care of your own needs. Then you’ll want to think how much you want to leave to heirs—both through annual exclusion giving during life and at death. Many families worry if they leave “too much,” their heirs won’t have the same incentive to find meaningful work whether that be a traditional job or charitable work.

More and more, clients are asking us to evaluate how much they can afford to give to causes that touch their hearts. Once we know there is a charitable intent, we can think about which assets make the most sense to donate. We can also evaluate different types of charitable vehicles, like charitable remainder trusts or charitable lead trusts or donor advised funds, to see what makes the most sense in each situation.

Another “aha” moment may be a discussion with your advisor about just how much risk you need or want to take. If you have “enough” money to meet your goals with a lower overall target return, then you may be able to reposition your portfolio to reduce your exposure to riskier assets. After a long bull market, this may be a good time to have that conversation. It’s tricky though when taxes are involved, so think it through carefully.

Legacy Considerations
As you reach your later years in retirement, there are more considerations. Retirement assets can be one of the least “efficient” assets to leave heirs. You may want to change your retirement beneficiary designations to your donor advised fund, or other charitable vehicle, if you have plenty of assets in other types of accounts to leave heirs.

If one spouse is in bad health, you may want to think about moving highly appreciated assets to the least healthy spouse so that the surviving spouse gets a step-up in cost basis at the first death. I know this is not an easy thing to think about, so once again your advisor may be able to think it through with you.

Some people may worry about running out of assets. We’ve seen situations where a reverse mortgage may help you to stay in your house longer. These vehicles are only available once you are over age 62. Don’t wait too long to put them in place. Even high net worth individuals are using reverse mortgages as a way to leverage home equity–especially should we see a downturn in the stock market.

One of the great joys we see with clients is the ability to give money away while they are still alive. Your advisor can help you determine if this makes sense in your situation. In some cases, you can use a family trust to pay out a stream of income to a child to help them live a more comfortable lifestyle now and not have to wait for an inheritance.

When you give money to heirs, they don’t have to pay gift tax. You may not either if you haven’t used up your lifetime gift tax exemption. With the last tax law change, you can give up to $11.2 million per person without gift or estate tax in your lifetime or at death. Those thresholds could be lowered in the next few years, so you may want to think about giving soon.

If you want to make charitable donations, more people are using donor advised funds. They are easy to set up and fund. Appreciated stock may be one of the best ways to transfer assets to your favorite charities while avoiding paying capital gains tax on the sale of appreciated stock.

Summary
Financial planning is a moving target throughout life. Weaving in tax issues while making smart investment decisions can save millions over time. But it’s really all about using your money to support the life you want to live and finding joy in helping others along the way.

Disclosures
The information presented here is not specific to any individual’s personal circumstances. To the extent that this material concerns tax matters, it is not intended or written to be used, and cannot be used, by a taxpayer for the purpose of avoiding penalties that may be imposed by law. Each taxpayer should seek independent advice from a tax professional based on his or her individual circumstances. These materials are provided for general information and educational purposes based upon publicly available information from sources believed to be reliable—we cannot assure the accuracy or completeness of these materials. The information in these materials may change at any time and without notice.