Post

Machine learning algorithms such as neural networks are supposed to have some sort of universal uniform approximation theorem that shows that they can (at least in principle) learn any possible data set without simply overfitting to the training data.

The standard universal approximation theorem applies to shallow neural networks with arbitrary continuous non-polynomial activation functions. There are also plenty of polynomial approximation results in mathematics, so one should at least in principle be able to train a real multivariate polynomial to model an arbitrary continuous function arbitrarily well using a multi-layered machine learning model. On the other hand, anyone modestly familiar with complex analysis knows that not every continuous function from a compact subset of C to C can be approximated by polynomials.

In this post, we shall eventually produce a work-around that allows us to approximate arbitrary continuous functions using complex polynomials.

Why use polynomials instead of deep neural networks?

I personally have many issues with neural networks. To me, neural networks are clumsy to study mathematically. For example, a deep neural network with tanh activation is far more complicated than the function f ( x ) = tanh ( tanh ( tanh ( x ) ) ) , but the function f is awkward to work with. For example, tanh ( x ) can be written in terms of the exponential function, but this means that f ( x ) is written in terms of an iterated exponential. I personally would rather not use iterated exponentials. To make things worse, f ( i ⋅ x ) = i ⋅ tan ( tan ( tan ( x ) ) ) which has a complicated singularity set. The unbiased neural networks with tanh activation essentially become neural networks with tan activations when fed purely imaginary inputs, and neural networks with tangent activations are untrainable and pathological. Neural networks with ReLU activations do not have this pathology that we see with tanh activation, but they have their own pathologies.

It seems like the pathological attributes and behavior of neural networks are part of the reason they are so difficult to interpret and study mathematically, so an inherently interpretable alternative to neural networks may help us solve the problems related to AI interpretability and safety. Of course, inherently interpretable AI should ideally match or at least complement the performance of deep neural networks since we need inherently interpretable AI to be relevant to AI safety.