Working Up

by Dwayne Phillips

Somewhere, somehow we forgot what a good approximation was.

We used to tell this “joke” a long time ago in a place far, far away.

Consider the situation with an amorous young gentlemen on one side of a room and an amorous young lady on the other side of the room. Every two minutes these two persons reduce the difference between them by half. At no point in time will they ever touch one another.

The above has to do with the theory of limits in mathematics. It is true. The two persons will never touch given what is described.

The punchline of the joke is: Yes, but after a few iterations they are close enough for a good approximation. (No one ever claimed the joke was funny.)

Now back to reality…

Some folks at MIT discover that most neural networks have 10 or 100 times more neurons and layers than necessary. The persons building the neural networks have forgotten or have never heard about the good approximation. This is partly due to the near-zero cost of processors and other computing machinery. I believe it is mostly due to the lack of humor in higher mathematics courses taught in colleges today (we do still teach such courses, don’t we?).

Good approximations are often good enough. Especially when using data that isn’t that good. Consider the input quality and the needed and achievable output quality.