Older workers have been leaving the UK workforce because they want to wind down, are ill and have family responsibilities, but may end up less financially resilient as a result.