Determine your return function. Either get it from the math problem you're trying to solve, or calculate a function based on historical data of inputs and outputs. For example, you may have a parabolic return function of f(x) = -x^2 + 20x + 10.
Find the derivative of your function from Step 1. In the example, f'(x) = -2x + 20.
To find when additional inputs actually make overall output go down, maximize your function by solving the derivative in step 2 for 0.
0 = -2x + 20
x = 10
After 10 units of input, the output begins to go down instead of up.