Indeed there is a great preponderance of theory and opinion about the efficiency of markets. I remember reading Malkiel's book many years ago when I was reading everything I could find about investing. He seemed eloquent and educated but I don't recall what impression, if any, I had of his central thesis that markets are efficient and as a result nobody can consistently outperform the markets. Perhaps I missed it at the time or was unmoved. These days I reject that view wholeheartedly.
From a personal motivation standpoint, which should you allow to have a greater influence? Three great investors, Graham, Fisher and Lynch, who wrote books in great detail on their methods of investing which generated superior results for decades, or a book by a university prof explaining the theory of why this is impossible.
As it turns out, in the subsequent 20 years I have made a career out of the inefficiency of the markets as a stock picker. Maybe this has worked out because I have a very narrow focus or I am thorough in my work or I am good at pattern recognition or I am lucky. I make a good number of mistakes but I have also improved my method over the years so that my mistakes are less severe and less frequent.
But not once in the past 20 years have I ever stopped myself to say, hold on, Malkiel has proven in theory that what I do is not possible. I am pretty sure the same goes for other, more famous, stock pickers: Buffett, Lynch, Fisher, Gross, Druckenmiller, Klarman, Miller, etc.
Yes it is true that most investors, pros or otherwise, are not very good. This may be why many in the investment industry believe in the efficient market theory. It provides a good excuse for their poor performance. To go back to golf, do hacks look at the collapse of Tiger Woods' career and say that this proves that nobody can sustain a superior result in the game over the long term?
I have met only a handful of very smart institutional investors with long-term track records well above anything the random walk would predict, say 17-20% annual returns for 20 years or more. But the best investor I ever met is an individual, a private investor, who has turned his own small money into somewhere between $40m-140m over a decade or so. He started on his own just a smart guy with a unusual insight into mathematics and cash flow analysis. He pursues the independent research angle a little bit but shyness and a lack of industry expertise get in the way. His real angle is an almost genius ability to interpret financial statements and to uncover hidden jewels trading at abnormally low prices. I have watched him do this repeatedly.
There is a theory to support every point of view. But I would encourage people to aim to achieve what others consider impossible.