I did not understand your question but I will try to answer to the two possible meanings of your answer.
I am wondering why we use this technique instead of finding the lowest line which has the lowest variance
Here, I understand that you want to ask why is not a "minimum variance" method used. The equation of the "line" with minimum variance is always a straight line parallel to the $x$ axis -the mean- (variance = 0).
Is there a difference between averaging the sum of squared difference and not averaging it?
Here, I understand that you want to get rid of the averaging term of the equation. You can get rid of it without any problem, it's a constant and does not have any influence in the result.
Why would we not just get rid of the least squares method and just find variance to find line of best fit?
The reason of why we don't get rid of it is because is the only differentiable method to minimize the error between the regression and the data, other methods exist but are skewed estimators