ASK what was holding me back and I can tell you immediately: fear. Fear of failure, in fact. This self-fulfilling mental condition is a disaster for too many careers.

I wrote What’s Stopping You? out of frustration with this and the self-help universe. I saw a disconnect between what the psychologists stated were our innate fears and the life-changing promises of the self-help gurus. Instead, I thought it important to help us understand our insecurities, to accept them as part of us, and to help navigate the consequences.

According to psychological experiments on children, around a quarter of the population suffer from some form of fear-related insecurity – many of them unaware that it’s their fears that are disabling their career prospects. Many may mask their fears via rebelliousness or being the office clown – or stating they are far happier “among the troops than the officers”. Others may blame barriers such as prejudice or poor luck. They can become part of the moaning canteen gang – often criticising or even feuding with colleagues. All are a disaster for modern careers – and all are a mask for an inner sense of inadequacy brought about by fear.

A key problem is that our fears are self-fulfilling. Take my first book – the Pursuit of Happiness, published in 2000. It was on my New York banking career and came about after I won a highly-respected literary agent and a deal with a major publishing house. However, the book sold less than my (unrealistic) expectations, which left me dejected and depressed. I judged it as a total failure and behaved accordingly. Yet the book could have been seen as a strong first step as an author – it was my determination to see it as a failure that led others to also judge it as such. In fact, they were mostly judging my responses.

So how did such a debilitating condition come about? Experiments on 1960s American children – especially those by John W Atkinson at Stanford University – divided children into those that had high “achievement motivation” (High-AMs as I call them in What’s Stopping You?) and those that had high fear of failure (High-FFs). In his 1960 study, written up in the Journal of Abnormal and Social Psychology, Atkinson offered the children a range of reward-based tasks. High-AMs approached them expecting success and were focused on the pay-off success would inevitably bring. High-FFs, meanwhile, expected failure and became anxious about even intermediately-difficult tasks. In many cases, they sought to avoid or even sabotage the task.

Further studies – especially those by Dennis Charney – concluded that our fears could be mild post-traumatic stress disorder – perhaps caused by early-life traumas, especially (with respect to fear of failure) those involving episodes of public humiliation. These episodes have generated a “fear conditioning” that can be triggered by a – perhaps unconscious – reminder of the original episode. What’s more, this is a hardwired condition: once afflicted, our first response is always likely to be based on our fears.

For me, a key problem was my workplace relationships. I had developed a deep suspicion of both seniors and peers caused by my family divisions when a small child. This manifested itself in envy, distrust and a fear of rejection – all of which were my instant responses to workplace politics. Of course, they were irrational, but they were also self-fulfilling – preventing me developing a more rational second response, not based on my fears.

Yet there is hope – although not via the “quick-fix” cures of the modern self-help industry. Techniques such as hypnotism or acupuncture are no more than a form of denial in my opinion, setting us up for a potentially damaging reckoning further down the line, perhaps when we hit a major barrier. Certainly, for me, I was left depressed and defeated by the failure of my first book, despite having read plenty of self-help books that should have prepared me for such a disappointment.

My route involves, first, fully understanding our condition. We then need to accept it as part of us – not deny it through some self-help inspired charge in a different direction. And finally we need to navigate the consequences: realising that our first response may have been inappropriate (perhaps triggered by our fears) and forcing us to look for a more rational second response – hopefully close enough to the original response to stop its damage.

Robert Kelsey: What’s Stopping You? Why Smart People Don’t Always Reach Their Potential and How You Can, is available in WHSmiths priced £10.99.