William Safire of The New York Times says the term was invented by Arthur Conan Doyle, the creator of Sherlock Holmes. The term "smoking pistol" appeared in the Sherlock Holmes story "The Gloria Scott." A murder takes place aboard a prison ship. The murderer is found standing over the victim while holding a smoking gun. Elementary, dear Watson. Today's guns don't "smoke," but they did before smokeless gunpowder was invented. In Sherlock Holmes' case of the Gloria Scott, the smoke coming from the gun barrel indicates that the gun has just been fired.
Urban Dictionary defines a smoking gun as an irrefutable piece of evidence. Such evidence is often critical in a court trial. If a criminal was actually holding a gun at a crime scene, it would be almost undeniable evidence of criminal activity. In white collar crime, smoking gun evidence is damning emails or a trail of banking activity that shows evidence of fraud. Mass graves are smoking gun evidence of genocide. When smoking gun evidence emerges, it's often impossible to argue against.
A person might say, "I know Mary stole all my tomato plants because I caught her in the garden holding the vines in her hands." In this case, the vines become the smoking gun evidence. A more literal example: "I know Ed shot Carla because when I walked into the living room, he was holding a gun and she was dead on the floor."
The Watergate scandal in the 1970s led to the resignation of President Nixon. Nixon had a tape recording system in his office. He recorded conversations which proved that he tried to cover up the investigation of the break-in at the Democratic National Committee headquarters--the start of the Watergate scandal. The U.S. Supreme Court ruled that the President must hand over the tapes. The tapes became known as The Smoking Gun because they provided evidence that Nixon covered up evidence of the break-in. Representative Barber Conable, a conservative Republican, stated that the evidence provided by the tapes looked "like a smoking gun." After Watergate, the term smoking gun gained popularity.
Smoking gun also refers to a popular website by the same name. The Smoking Gun posts damning evidence on its website like mugshots, arrest records and legal documents. Their material is obtained from government and law enforcement sources. The Smoking Gun's goal is to bring information to the public--information unreported by mainstream media.