2
$\begingroup$

I am working on building a scoring algorithm for student data,

say the attributes are :

name, location, age, class, school_name, skill1, skill2, skill3

based on these data I need to create a student score.

I need to assign weight-ages for age, class, school_name skills and come up with a score for student.

say I have 2 scoring models like :

score_1 = x1*location_weight + x2*age_weight + x2*class_weight + x3*school_name_weight + x4*skill1_weight + x5*skill2_weight + x6*skill3_weight

score_2 = y1*location_weight + y2*age_weight + y2*class_weight + y3*school_name_weight + y4*skill1_weight + y5*skill2_weight + y6*skill3_weight

now how can I compare these models and evaluate them?

The problem is I don't have a test or validation set to prove or compare how accurate each of these model is, so in this case what is the best approach to compare and validate different models? also what is the best ways to build a validation set from scratch?

Improve this question
$\endgroup$
  • $\begingroup$ What do you think the 'model' should predict? Getting some certain score by student in future? $\endgroup$ – IharS Sep 23 '15 at 9:13
  • $\begingroup$ a score in range of 1-100, as a measure of student ranking $\endgroup$ – Sreejithc321 Sep 23 '15 at 9:38
  • $\begingroup$ You want to predict score change in time? $\endgroup$ – IharS Sep 23 '15 at 9:58
  • $\begingroup$ no, just give a score based student data. so i have the above mentioned attributes and based on that how can I build a model to predict student score ? also how to validate the score predicted by that model ? $\endgroup$ – Sreejithc321 Sep 23 '15 at 11:11
4
$\begingroup$

Predicting and scoring are two different tasks.

And according to your answers and comments you are not solving prediction problem. You just want to set to each student a number in range [1,100] according to some rule. This is ranking (or scoring, whatever).

Therefore, the terms #prediction_model, #accuracy, #validation, #training_set are out of this scope. You don't need to validate anything. You are not making predictions.

What you want is to map ranks to students.

But a problem is that you have mostely categorical data (school name, location etc) that cannot be 'ranked'. Some of them are useless at all: how does the student name refer to his school progress? :)

If you change it somehow to numerical (e.g. 'Skill_1_level', 'Skill_2_level', 'remoteness_of_location', 'school rank' etc) than you can do some ranking:

  1. Normalize data: each of your factors

enter image description here

  1. Multiple by 100, as you want [0,100] range instead of [0,1]
  2. Set up weights based on your experience according to factor's importance. So that the sum of weights is 1.
  3. And finally build a rank (score):

Rank = 0.1 * skill_1_level + 0.2 * skill_2_level + 0.05 * remoteness_of_location + 0.5 * school_rank + ...

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks for the answer, So in this case need to validate the weigtages used for each attributes in my scoring model in order to prove how effective this ranking model is. correct ? for that I don't have any valid dataset, all weightages are assigned based on intuition. $\endgroup$ – Sreejithc321 Sep 24 '15 at 6:59
  • $\begingroup$ First of all - define what is effective. Once again: You have data set. A set of students and their data. You can devide it into training and validation sets. But which model are you going to train? What do you want to predict ? $\endgroup$ – IharS Sep 24 '15 at 8:23
  • $\begingroup$ You can tune your weights, model it. But target variable should be defined: for best weights selection the model must know which parameter it optimises. $\endgroup$ – IharS Sep 24 '15 at 8:30
-1
$\begingroup$

I would start by fixing a seed (so that results are reproducible), then selecting a subsample (say, 10pct of data) and saving that as validation set. If you're using R, check the caret package

https://cran.r-project.org/web/packages/caret/index.html

and if Python is your weapon of choice, familiarize yourself with the sklearn documentation in 3.1:

http://scikit-learn.org/stable/modules/cross_validation.html

Both of those have multiple functions that might come in handy. If your computational resources allow it, I would suggest a cross-validation procedure (evaluating a model on multiple training-validation splits), both of the links I gave contain references to multiple functions that can help you achieve that.

Improve this answer
$\endgroup$
  • $\begingroup$ But the issue is there is no validation/test data to compare or measure error. Is there a way to build validation data ? $\endgroup$ – Sreejithc321 Sep 23 '15 at 9:41
  • $\begingroup$ You can't build this data. Collect it. And then devide into training and validation data sets. $\endgroup$ – IharS Sep 23 '15 at 10:10
  • $\begingroup$ I meant: take your existing training set, split it, keep one part as actual training and the other as validation. If you do cross-validation, you are sure to use all the data in either function at some point. $\endgroup$ – kpb Sep 23 '15 at 10:49
  • $\begingroup$ score is not availabe to calculate accuracy.I need to predict 'score'. $\endgroup$ – Sreejithc321 Sep 23 '15 at 11:07
  • $\begingroup$ @ IharS yes, that is my question. How to make this validation/testing data ? $\endgroup$ – Sreejithc321 Sep 23 '15 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.