I think the mistake Vito made was using Luca as his spy. First of all Luca's loyalty to the Don was legendary, and I don't think Sollozzo or Tattaglia Jr. bleieved the deception for one minute. After all Luca was not exactly the smoothest character around. If anything, by sending Luca to pretend he would betray Vito showed Sollozzo, Tattaglia and Barzini how seriouisly Vito was taking his "no." It also told them that Vito would not simply allow the drug business to flourish as long as Sollozzo's interests did not conflict with his. in other words the real message Luca inadvertantly sent was that Vito had to be taken out.