Somewhere in elementary school, you probably learned that the sum of the angles of a triangle equals 180 degrees. This is true if you are operating on a flat surface, but when you turn that surface into a curved shape, the rules of geometry change with it. Euclidean geometry is so named because it obeys Euclid's fifth postulate, the parallel postulate. In non-Euclidean geometry, the parallel postulate does not hold. There are two types of non-Euclidean geometry in three-dimensional space: hyperbolic and elliptic. In hyperbolic geometry, the sum of the angles of a triangle is less than 180 degrees. There are no similar triangles, and given a line L and a point P, there are infinitely many lines parallel to L that pass through P. In elliptic geometry, on the other hand, there are no parallel lines, and the sum of the angles of a triangle is greater than 180 degrees.
The concept of infinity is difficult for humans to grasp, but in advanced mathematics, you can learn about different types of infinity, how to multiply infinity or even raise infinity to the power of infinity. Infinity factors into several advanced mathematical concepts, such as set theory, which studies the mathematics of sets of numbers, and topology, which investigates the properties of objects that are preserved even when the object is deformed in special ways. You can also explore paradoxes related to infinity, such as Hilbert's Grand Hotel and Xeno's Arrow.
Game theory is a mathematical attempt to analyze and predict the decisions of rational, strategizing agents. Psychologists sometimes call it the theory of social interactions. This theory applies to several different areas of study, particularly economics, engineering, biology, business management, political science, philosophy and computer science. There are several different types of "games" that a game theorist can examine: cooperative of a non-cooperative, individual or group, zero-sum or non-zero-sum, symmetric or asymmetric, and discrete, continuous or infinite. Game theorists can also calculate how agents will react when the rules of the game are changed.
In nature, there are some objects that can be broken down into tinier and tinier pieces, and each piece will look like a replica of the larger whole. These objects are called fractals, and mathematicians can create and formulaically describe these curious shapes. Broccoli, snowflakes, ferns, river systems, blood vessels and mountain ranges are all examples of natural fractals. Fractals generally require non-Euclidean geometry, and they possess strange topological dimensions -- not normal dimensions such as 3-D or 2-D, but in-between dimensions such as 2.6-D or 1.4-D. They're part of a relatively new branch of math, at less than 100 years old, and research is still being done on them. The possibilities for their application are not fully understood, though they have been used to study the dynamics of fluids and solve engineering problems.