Polynomial features

Polynomial features#

The modelling tools included in ISLP allow for construction of orthogonal polynomials of features.

Force rebuild

import numpy as np
from ISLP import load_data
from ISLP.models import ModelSpec, poly

Carseats = load_data('Carseats')
Carseats.columns

Index(['Sales', 'CompPrice', 'Income', 'Advertising', 'Population', 'Price',
       'ShelveLoc', 'Age', 'Education', 'Urban', 'US'],
      dtype='object')

Let’s make a term representing a quartic effect for Population.

quartic = poly('Population', 4)

The object quartic does not refer to any data yet, it must be included in a ModelSpec object and fit using the fit method.

design = ModelSpec([quartic], intercept=False)
ISLP_features = design.fit_transform(Carseats)
ISLP_features.columns

Index(['poly(Population, degree=4)[0]', 'poly(Population, degree=4)[1]',
       'poly(Population, degree=4)[2]', 'poly(Population, degree=4)[3]'],
      dtype='object')

Compare to R#

We can compare our polynomials to a similar function in R

%load_ext rpy2.ipython

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[5], line 1
----> 1 get_ipython().run_line_magic('load_ext', 'rpy2.ipython')

File /opt/hostedtoolcache/Python/3.12.15/x64/lib/python3.12/importlib/__init__.py:90, in import_module(name, package)
     88             break
     89         level += 1
---> 90 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1387, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1360, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1310, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:488, in _call_with_frames_removed(f, *args, **kwds)

File <frozen importlib._bootstrap>:1387, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1360, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1324, in _find_and_load_unlocked(name, import_)

ModuleNotFoundError: No module named 'rpy2'

We’ll recompute these features using poly in R.

%%R -i Carseats -o R_features
R_features = poly(Carseats$Population, 4)

np.linalg.norm(ISLP_features - R_features)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[7], line 1
----> 1 np.linalg.norm(ISLP_features - R_features)

NameError: name 'R_features' is not defined

Underlying model#

If we look at quartic, we see it is a Feature, i.e. it can be used to produce a set of columns in a design matrix when it is a term used in creating the ModelSpec.

Its encoder is Poly(degree=4). This is a special sklearn transform that expects a single column in its fit() method and constructs a matrix of corresponding orthogonal polynomials.

The spline helpers ns and bs as well as pca follow a similar structure.

quartic

Feature(variables=('Population',), name='poly(Population, degree=4)', encoder=Poly(degree=4), use_transform=True, pure_columns=False, override_encoder_colnames=True, spec=('poly', ('Population',), (('degree', 4), ('intercept', False), ('raw', False))))

Raw polynomials#

One can compute raw polynomials (which results in a less well-conditioned design matrix) of course.

quartic_raw = poly('Population', degree=4, raw=True)

Let’s compare the features again.

design = ModelSpec([quartic_raw], intercept=False)
raw_features = design.fit_transform(Carseats)

%%R -i Carseats -o R_features
R_features = poly(Carseats$Population, 4, raw=TRUE)

np.linalg.norm(raw_features - R_features)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 np.linalg.norm(raw_features - R_features)

NameError: name 'R_features' is not defined

Intercept#

Looking at py_features we see it contains columns: [Population**i for i in range(1, 4)]. That is, it doesn’t contain an intercept, the order 0 term. This can be include with intercept=True

quartic_int = poly('Population', degree=4, raw=True, intercept=True)
design = ModelSpec([quartic_int], intercept=False)
intercept_features = design.fit_transform(Carseats)

np.linalg.norm(intercept_features.iloc[:,1:] - R_features)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[14], line 1
----> 1 np.linalg.norm(intercept_features.iloc[:,1:] - R_features)

NameError: name 'R_features' is not defined