Even without reading the novel, it is easy to see how Michael was able to conclude that Tessio turned traitor. For one thing, Tessio openly questioned Michael in the meeting with Vito, Tom, Clemenza, Carlo, & Michael. Tessio boldly stated "Mike, you're wrong." He didn't believe that Michael was making the move out of strength, but out of weakness. In addition, Vito told Michael that someone close would betray him.
In the novel, it's stated that Vito ordered the phones of Tessio & Clemenza to be monitored.