...you'd be a cyclops. Refs make mistakes. Always. All the time, every game, no matter how good they are. When you ask one person to completely police a game, they are going to mess up. Our current system of self-officiation plus observers is incredibly accurate. Even the worst games are called more accurately than the best reffed games.
Let me define 'accurate.' In this situation, what I mean is whether or not the outcome of an individual call is correct. Is the 'travel' a travel? Is a 'foul' a foul? In ultimate, the answer is yes and yes and yes. In reffed sports, the answer is sometimes yes and sometimes no.
Accuracy hides a much bigger issue: fairness. Each individual call may be correct, but if one team is systematically calling a lot of stuff that the other team isn't, you end up with an unbalanced and unfair game. This happens all the time in reffed games. The difference in ultimate is that it is the competitors themselves manufacturing the advantage. This issue of fairness and the ability of one team to create an advantage for themselves is the strongest argument for refs.
A few thoughts on this. First, adding refs won't increase accuracy - it will decrease it. It won't increase fairness, either. It will actually create more games that are decided unfairly. All it will do is remove the control of that unfairness from the players. Finally, don't be confused into thinking that adding refs will keep people from working the rules to their advantage. On the contrary, adding refs will greatly increase the amount of cheating and gamesmanship as players and coaches learn to really take advantage of the rules, enforcement and the officials.