I have physical measurements with length 2*n, where the first vector represents a charge or a capacity (in Coulomb) $C$ and the second one is a voltage $V$. Let's call this measurement "forming". A measurement consist in increasing the charge until a voltage threshold is reached. I want to predict the rank of a sample depending on the these physical measurements. A rank is a letter: A, B, C... However the physical measurements have different lengths ($n\in [1000,3000]$). Besides this, the voltage thresholds are the same for all samples: $V_a[i] = V_b[i]$ with $a,b$ being any sample number and $i\in {0,-1}$ (first and last value in python notation). So we can say that the $V$ vectors are already normalized (same mean and std) and can be used as indexes.

Determining the rank can be done with a similar measurement but is too long, that is why I want to establish a predictive model. Let's call this other long measurement "determining". I have used "PCA" so far, which requires same-length vectors to make a matrix.

This is my plan:

  • make the "forming" measurement for 100 samples (done)
  • establish the rank for 100 samples with the "determining" measurement (done)
  • develop a predictive model
  • use the model on the next 100 000 samples

What I would like to know:

  • your opinion on whether my handling of heterogeneous length is correct
  • Is PCA the right tool for the model?

What I have done so far:


from matplotlib import pyplot as plt
import numpy as np
from numba import jit
import pandas as pd
from sklearn.decomposition import PCA as sklearnPCA
from scipy.signal import savgol_filter

get data

logs = ... #list of 100 pandas.DataFrame
Ranks = ... #list of corresponding 100 ranks (e.g. characters)

As the $C$ vs $V$ function is not linear but has plateaus and walls, I cannot simply use a fixed sampling rate for each sample to resize to a fixed length. Otherwise, I will have missing data on the "walls". Hence I'm using on the first sample the 'resized' function of B.M. which both smooths and resizes by averaging neighboring data points, then I use a Savitzky-Golay filter for smoothing and a linear interpolation for resizing, using as target for the interpolation the $V$ points found by the resized function on the first sample:

#%% resizing functions
def resized(data,N):
    for n in range(N):
        summ = carry
        while m*N - n*M < M :
            summ += data[m]
            m += 1
        carry = (m-(n+1)*M/N)*data[m-1]
        summ -= carry
        res[n] = summ*N/M
    return res
resized = jit(resized)

def res_interp(xnew,xp,yp):
    yp = savgol_filter(yp,7,2)#smooth with a 7-point window
    return np.interp(xnew,xp,yp)

#%% make the same length
for i,l in enumerate(logs):
    ch1=l[l.Step==2]#get only correct measurements

    if i==0:#base sampling on first data



charge1 is a $1000\times100$ matrix where rows correspond to 1000 voltages between $V[0]$ and $V[-1]$ (in charge1.index) and columns represent the charges of the 100 samples.

sklearn_pca = sklearnPCA(n_components=4)#I picked 4 for readability
sklearn_transf = sklearn_pca.fit_transform(charge1.Capacity.values)
Rank_int = [ranks.find(x) for x in Rank]#string to int
compo = sklearn_pca.components_
# plots
fig, ax = plt.subplots(nrows=1)
fig, ax = plt.subplots(nrows=2,ncols=2)
ax = ax.flatten()
for i in range(compo.shape[0]):

The plots of charge1.index versus PC1 is not so interesting: all the curves $C$C versus $V$ have a similar shape, so PC1 is simply an average of all of them. I believe more interesting are the "scores" given to the following PCs which are uncorrelated to PC1. I see still no correlation which means I might not be using the right tool or my resizing made my data wrong:


I fell on a sample of samples with no Rank A so the experiment was certainly not optimal, but I think my method is more problematic than my samples.

side note: I have seen so far little examples of "real life" measurements with noise and heterogeneous lengths. Most of them are either number recognition or face recognition of fixed-length images, without counting stock market analysis which is a very different experimental set-up (there are no "sample", only continuous measurements).

edit: I believe I need to use from sklearn.ensemble.GradientBoostingClassifier and not PCA. I saw using PCA on physical measurements in a lecture given by ORNL (Manhattan project place) in order to extract "hidden" physical behaviors (PCi, i>2) behind the main physical feature (PC1) that could correspond to different materials properties. But it seems it's not the right tool as it's used by real data scientists to simplify a multi dimensional problem. Any input would still be appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.