1
$\begingroup$

I have physical measurements with length 2*n, where the first vector represents a charge or a capacity (in Coulomb) $C$ and the second one is a voltage $V$. Let's call this measurement "forming". A measurement consist in increasing the charge until a voltage threshold is reached. I want to predict the rank of a sample depending on the these physical measurements. A rank is a letter: A, B, C... However the physical measurements have different lengths ($n\in [1000,3000]$). Besides this, the voltage thresholds are the same for all samples: $V_a[i] = V_b[i]$ with $a,b$ being any sample number and $i\in {0,-1}$ (first and last value in python notation). So we can say that the $V$ vectors are already normalized (same mean and std) and can be used as indexes.

Determining the rank can be done with a similar measurement but is too long, that is why I want to establish a predictive model. Let's call this other long measurement "determining". I have used "PCA" so far, which requires same-length vectors to make a matrix.

This is my plan:

  • make the "forming" measurement for 100 samples (done)
  • establish the rank for 100 samples with the "determining" measurement (done)
  • develop a predictive model
  • use the model on the next 100 000 samples

What I would like to know:

  • your opinion on whether my handling of heterogeneous length is correct
  • Is PCA the right tool for the model?

What I have done so far:

imports

from matplotlib import pyplot as plt
import numpy as np
from numba import jit
import pandas as pd
from sklearn.decomposition import PCA as sklearnPCA
from scipy.signal import savgol_filter

get data

logs = ... #list of 100 pandas.DataFrame
Ranks = ... #list of corresponding 100 ranks (e.g. characters)

As the $C$ vs $V$ function is not linear but has plateaus and walls, I cannot simply use a fixed sampling rate for each sample to resize to a fixed length. Otherwise, I will have missing data on the "walls". Hence I'm using on the first sample the 'resized' function of B.M. which both smooths and resizes by averaging neighboring data points, then I use a Savitzky-Golay filter for smoothing and a linear interpolation for resizing, using as target for the interpolation the $V$ points found by the resized function on the first sample:

#%% resizing functions
def resized(data,N):
    M=data.size
    res=np.empty(N,data.dtype)
    carry=0
    m=0
    for n in range(N):
        summ = carry
        while m*N - n*M < M :
            summ += data[m]
            m += 1
        carry = (m-(n+1)*M/N)*data[m-1]
        summ -= carry
        res[n] = summ*N/M
    return res
resized = jit(resized)

def res_interp(xnew,xp,yp):
    yp = savgol_filter(yp,7,2)#smooth with a 7-point window
    return np.interp(xnew,xp,yp)

#%% make the same length
charge1=list()
for i,l in enumerate(logs):
    ch1=l[l.Step==2]#get only correct measurements

    if i==0:#base sampling on first data
        y_ch1=resized(ch1.Capacity.values,1000)
        x_ch1=resized(ch1.Volt.values,1000)
    else:
        y_ch1=res_interp(x_ch1,ch1.Volt.values,ch1.Capacity.values)


    charge1.append(pd.Series(data=y_ch1,
                 index=x_ch1,
                 name='Capacity'))

charge1=pd.concat(charge1,axis=1)

charge1 is a $1000\times100$ matrix where rows correspond to 1000 voltages between $V[0]$ and $V[-1]$ (in charge1.index) and columns represent the charges of the 100 samples.

sklearn_pca = sklearnPCA(n_components=4)#I picked 4 for readability
sklearn_transf = sklearn_pca.fit_transform(charge1.Capacity.values)
ranks='ABCD'
Rank_int = [ranks.find(x) for x in Rank]#string to int
compo = sklearn_pca.components_
# plots
fig, ax = plt.subplots(nrows=1)
ax.plot(charge1.index,(sklearn_transf[:,0]-sklearn_transf[:,0].mean())/sklearn_transf[:,0].std())
ax.plot(charge1.index,(sklearn_transf[:,1]-sklearn_transf[:,1].mean())/sklearn_transf[:,1].std())
ax.plot(charge1.index,(sklearn_transf[:,2]-sklearn_transf[:,2].mean())/sklearn_transf[:,2].std())
ax.plot(charge1.index,(sklearn_transf[:,3]-sklearn_transf[:,3].mean())/sklearn_transf[:,3].std())
fig, ax = plt.subplots(nrows=2,ncols=2)
ax = ax.flatten()
for i in range(compo.shape[0]):
    ax[i].plot(Rank_int,compo[i,:],'o',label=str(i))
    ax[i].set_xticks([1,2])
    ax[i].set_xticklabels(['B','C'])
    ax[i].set_title('PC'+str(i+1))

The plots of charge1.index versus PC1 is not so interesting: all the curves $C$C versus $V$ have a similar shape, so PC1 is simply an average of all of them. I believe more interesting are the "scores" given to the following PCs which are uncorrelated to PC1. I see still no correlation which means I might not be using the right tool or my resizing made my data wrong:

Eigenvalues

I fell on a sample of samples with no Rank A so the experiment was certainly not optimal, but I think my method is more problematic than my samples.

side note: I have seen so far little examples of "real life" measurements with noise and heterogeneous lengths. Most of them are either number recognition or face recognition of fixed-length images, without counting stock market analysis which is a very different experimental set-up (there are no "sample", only continuous measurements).

edit: I believe I need to use from sklearn.ensemble.GradientBoostingClassifier and not PCA. I saw using PCA on physical measurements in a lecture given by ORNL (Manhattan project place) in order to extract "hidden" physical behaviors (PCi, i>2) behind the main physical feature (PC1) that could correspond to different materials properties. But it seems it's not the right tool as it's used by real data scientists to simplify a multi dimensional problem. Any input would still be appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.