Speaking moments after meeting the ousted leader, he said Mr Gachagua was recovering.
Have you become this heartless, Mr President? Cleophas Malala
Since last night, they and a group of political leaders from Azimio la Umoja are the first politicians allowed to see the immediate deputy. Others were Eugene Wamalwa and Senator Joe Nyutu.
"We are privileged that some of us were allowed to see him. Myself, Wamalwa and Nyutu saw him. We wish Gachagua a speedy recovery. We thank God for sparing his life,” Mr Musyoka said.
The Senators proceeded to impeach him while he was in hospital. But Mr Kalonzo poked holes in the developments, especially after Mr Gachagua was admitted to hospital.
"The law of natural justice demands that no person should be convicted without being heard. The DP team will argue that out in Court," said Musyoka before adding, "Let justice reign in our courts."
The impeachment of the deputy president under the 2010 Constitution is in many ways unprecedented. It is a test of the Constitution in every sense.
"These things are important parts of our history and heritage as a people," said Mr Musyoka.
But according to the Wiper Party leader, what happened in the Senate last night, historic as it was, had "serious" legal flaws.
"But truth be told, it smacks of gross injustice [and] political manipulation," Mr Musyoka argued.
The law of natural justice demands that no person should be convicted without being heard, Mr Kalonzo said, adding that Mr Gachagua's legal team would be able to argue this in court.
The Senate found Mr Gachagua guilty on five of the 11 counts on which the National Assembly impeached him.
Parliament votes on the nomination of CS Kindiki as new DP
For his part, DAP-K party leader Eugene Wamalwa said Thursday night, when the Senate upheld Mr Gachagua's impeachment, was a sad day for the country.
The DAP-K leader praised Mr Gachagua for choosing to "face his accusers on his feet and not on his knees".
"If we ever doubted that our parliament is under the control of the executive, last night was proof," Mr Wamalwa said, adding that it was clear that parliament was dancing to the tune of the executive.
He further poked holes in the neutrality of Mr Amason Kingi and Mr Moses Wetang’ula - both speakers of the Senate and National Assembly respectively.
"Why was the Senate in a hurry to crucify this man and bury him politically?" Mr Wamalwa said, adding that as far as the National Assembly was concerned, Mr Wetang’ula had openly and publicly exposed his bias.
"It is not all over. Our last line of defence is the judiciary," he said.
Commenting on last night's events, Lamu Senator Joseph Githuku said they had tried everything to save Mr Gachagua from impeachment but the die had already been cast.
