Place marks on the sides of the square. Each mark must be a specific distance from a corner. Compute the distance from the corners where the marks should be made by dividing the length of the side of the square by 3.414. For example, if the length of one side of the square is 36 inches, the distance from the corners where the marks should be placed is 36/3.414 = 10.545 inches. When you are finished marking, there should be two marks on each side of the square. Each mark should be 10.545 inches from a corner.
Connect the two marks nearest each corner with a line. When you are finished, each corner should look like a triangle. If you made the marks correctly, the lengths of the drawn lines should be equal to the length of the side of the square divided by 2.414. For the 36-inch square, the length of the lines should be 36/2.414 = 14.913 inches. If this distance is wrong, measure your marks again. You should get all the measurements correct before you do any cutting.
Cut along each line to get the octagon. If you did everything correctly, the length of each of the eight sides should be the same. The correct length for each side is the length of a side of the original square divided by 2.414. If you started with a 36-inch square, the length of each side of the octagon should be 36/2.414 = 14.913 inches.