All Questions

Filter by
Sorted by
Tagged with
16 votes
3 answers
5k views

How to self-learn data science? [closed]

I am a self-taught web developer and am interested in teaching myself data science, but I'm unsure of how to begin. In particular, I'm wondering: What fields are there within data science? (e.g., ...
xyhhx's user avatar
  • 263
3 votes
1 answer
2k views

Sampling with replacement, specify the probabilities

I am trying to do sampling with replacement in Scala/Spark, defining the probabilities for each class. This is how I would do it in R. ...
Stefano's user avatar
  • 31
4 votes
1 answer
495 views

Alternatives to Graph theory?

I was planning to use a graph theoretic and page-rank approach to finding the most influential person in an organization. Influential person is someone who drives a lot of activity in the ...
user avatar
3 votes
1 answer
116 views

Conditional logit in Stata

Question: Can Stata present or output the actual betas/coefficients in a datasheet when it performs the conditional logit model? I understand that Stata has a "predict" capability, but I need to ...
mdg's user avatar
  • 131
4 votes
2 answers
2k views

Handling many missing values

I have a dataset I want to perform multivariate linear regression to it. The dimensions of the dataset are 832085 rows and 11 columns. The data are quite messy and given the size and my lack of ...
Dr.Fykos's user avatar
7 votes
2 answers
8k views

SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?

I am using LogisticRegressionWithLBFGS to train a multi-class classifier. Is there a way to get the probability of all classes (not only the top candidate class) ...
Rami's user avatar
  • 594
1 vote
0 answers
261 views

Pig is not able to read the complete data

I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR. The operations on the data are simple (JOIN and STORE), but the data is not getting ...
shanky_thebearer's user avatar
12 votes
1 answer
4k views

What's the relationship between an SVM and hinge loss?

My colleague and I are trying to wrap our heads around the difference between logistic regression and an SVM. Clearly they are optimizing different objective functions. Is an SVM as simple as saying ...
Simon's user avatar
  • 1,026
3 votes
0 answers
109 views

Is it common to preprocess image data before sending it through a deep net? [duplicate]

I'm curious as how convolutional neural network are used in practice for object recognition. Is it common to perform data preprocessing before providing the data to the input layer ? If so, what types ...
Ivar's user avatar
  • 131
0 votes
1 answer
150 views

Why in deep learning or another AI technique python is used rather than other languages? [closed]

I want to know, why python? Instead of C#, Java, C\C++ and so on?
Sashko's user avatar
  • 13
1 vote
1 answer
248 views

Importance of variables in RandomForest in R

I'm using varImpPlot() to evaluate the importance of variables from my rf model, and I can't decide wich variable I have to delete from my model. I'm not sure about the diference between "...
Michael Elma's user avatar
6 votes
1 answer
576 views

Is it advisable to rerun LASSO multiple (2) times?

I have a dataset with large (4500 variables) and 3000 observations (n < p case). I am using LASSO (from the glmnet package in R) to reduce dimensionality of the problem. I have couple of questions. ...
user62198's user avatar
  • 1,091
2 votes
1 answer
119 views

How to analyse move kindness in python (Logistic regression, neural network, etc.)?

With a team of researchers we were given the assignment to make a scale for the move kindness (how inviting a room or place is for exercise--e.g., a gym). In order to get objective results we were ...
Lucas Crijns's user avatar
7 votes
1 answer
1k views

SPARK, ML: Naive Bayes classifier often assigns 1 as probability prediction

Hi I am using Spark ML to optimise a Naive Bayes multi-class classifier. I have about 300 categories and I am classifying text documents. The training set is balanced enough and there is about 300 ...
Rami's user avatar
  • 594
0 votes
2 answers
85 views

Logistic regression program does not give statistically correct result [closed]

I wrote a very simple and compact logistic regression program using theano. I am initialising my data randomly, and I have restricted the number of training steps to 1, so since the weights are also ...
user's user avatar
  • 1,993
11 votes
3 answers
3k views

Can regression trees predict continuously?

Suppose I have a smooth function like $f(x, y) = x^2+y^2$. I have a training set $D \subsetneq \{((x, y), f(x,y)) | (x,y) \in \mathbb{R}^2\}$ and, of course, I don't know $f$ although I can evaluate $...
Martin Thoma's user avatar
  • 18.8k
2 votes
1 answer
108 views

Regression problem - too complex for gradient descent

I try to predict temperatures values as function of time and different parameters. The temperature curve look like a "ramp" with some "gauss peaks" on regular intervals. So, I try to build a ...
Michael Hooreman's user avatar
3 votes
1 answer
2k views

Handling categorical features in Factorization Machines algorithm - Feature Hashing vs. One-Hot encoding

For solving a prediction problem I'm willing to use the Factorization Machines, a model that in addition to learning linear weights on features, learn a vector space for each feature to learn pairing ...
Serendipity's user avatar
3 votes
1 answer
307 views

Handling a feature with multiple categorical values for the same instance value

I have data in the following form: table 1 id, feature1, predict 1, xyz,yes 2, abc, yes table2 id, feature2 1, class1 1, class2 1, class3 2, class2 I could perform a one many join and train on the ...
sudeepgupta90's user avatar
1 vote
1 answer
97 views

Algorithm to construct similarity structure from hash lookup table

I have constructed a lookup table using locality-sensitive hashing for comparing nearly similar documents/records. If two records (columns) have the same hash value in a row, they are considered to be ...
DonDyck's user avatar
  • 113
-2 votes
1 answer
470 views

How should I analyze this data from reddit (sample text included) [closed]

I have downloaded some data to learn about machine learning and distributed computing. I used WinZip to uncompress a .bz2. Now I have text (opened in Notepad) which looks like this (this file is long, ...
Thomas Shera's user avatar
4 votes
1 answer
220 views

Do I need to include a squared and linear variable in a random forest to achieve a parabolic effect?

I'm using a random forest model. One of my independent variables almost certainly has a parabolic effect on the dependent variable. In a linear regression, I would include the variable as both linear ...
Jesse's user avatar
  • 41
5 votes
2 answers
2k views

How to update weights in a neural network using gradient descent with mini-batches?

[I've cross-posted it to cross.validated because I'm not sure where it fits best] How does gradient descent work for training a neural network if I choose mini-batch (i.e., sample a subset of the ...
Alex's user avatar
  • 757
5 votes
2 answers
291 views

Random Forest Regression. How to represent really long list of categories for processing

I'm trying to build a model to solve a regression task. Simplified, the data look like: ...
evgenii's user avatar
  • 151
2 votes
0 answers
149 views

"Trending" feature to predict number of views

I am working on a problem where I have access to a database with news articles, their publication date and the number of views they got 24hrs they got published. The objective is to be able to ...
matthew henry's user avatar
3 votes
2 answers
207 views

Can PyLearn do everything that Theano can?

Since PyLearn2 is build upon Theano, is it possible to do anything I can do in Theano in Pylearn2? For example, if I have some snippets of Theano code, can I run them as-is in Pylearn2, or would this ...
robert's user avatar
  • 131
3 votes
4 answers
3k views

Extremely dominant feature?

I'm new to datascience. I was wondering how one should treat an extremely dominant feature. For example, one of the features is "on"/"off", and when it's "off", none of the other features matter and ...
gunit's user avatar
  • 133
2 votes
0 answers
135 views

Latent Semantic Indexing False Positive Detection

I am using the Gensim LsiModel. I have a set of documents and fixed set of topics. Some of the documents are already categorized, others are not. The goal is to categorize the uncategorized documents ...
user avatar
33 votes
3 answers
45k views

Hypertuning XGBoost parameters

XGBoost have been doing a great job, when it comes to dealing with both categorical and continuous dependant variables. But, how do I select the optimized parameters for an XGBoost problem? This is ...
Dawny33's user avatar
  • 8,266
4 votes
2 answers
5k views

When to use linear or logistic regression?

Still I have no clear concept when I should choose linear OR Logistic regression. In addition, when I can say either linear or logistic regression performing better? In particular, I want to know ...
mmr's user avatar
  • 141
0 votes
1 answer
104 views

Should i pick this as a career? [closed]

I'm mainly interested in creative things, but also interested in the science behind viewership and marketing a piece of media. Think about having a stable career in data science, where do you think I ...
Altoban's user avatar
  • 111
2 votes
1 answer
1k views

Early-Stopping for logistic regression. Theano

I am trying to understand the code for the logistic regression on the official documentation, but I am struggling to understand the logic behind this code: ...
Mohamed Lakhal's user avatar
0 votes
1 answer
1k views

Stemming text in R

I am stemming my text data in R. I am using a solution proposed by Yanchang Zhao for the latest version of tm package but found this very slow. I am looking for an alternative way. ...
Hamideh's user avatar
  • 940
9 votes
1 answer
12k views

What is the difference between affinity matrix eigenvectors and graph Laplacian eigenvectors in the context of spectral clustering?

In spectral clustering, it's standard practice to solve the eigenvector problem $$L v = \lambda v$$ where $L$ is the graph Laplacian, $v$ is the eigenvector related to eigenvalue $\lambda$. My ...
felipeduque's user avatar
2 votes
0 answers
202 views

How to create Self learning data product

I am trying to build price recommendation solution for clients in a scalable manner. I have two choices as below. Professional service: Statistician involvement to build regression model or any ...
StatguyUser's user avatar
11 votes
3 answers
1k views

How do AI's learn to act when the problem space is too big

I learn best through experimentation and example. I'm learning about neural networks and have (what I think) is a pretty good understanding of classification and regression and also supervised and ...
FraserOfSmeg's user avatar
3 votes
0 answers
706 views

Central Probability Interval

So, I'm having problems learning for a data analysis lecture. The topic is the construction of the "central interval" as our lecturer calls it. The concept shall path the way to understand confidence &...
F. Ha's user avatar
  • 31
1 vote
1 answer
3k views

How to build the following the string in R?

I am trying to build a long string of the form.....\'name1\',\'name2\',\'name3\',..etc. How do I do it ? I am trying to do it using R ? By hand is impossible as ...
user62198's user avatar
  • 1,091
3 votes
1 answer
272 views

How do write the following piece of code in R?

I am trying to write a piece of code for R to fetch data from an api with the following command: ...
user62198's user avatar
  • 1,091
2 votes
1 answer
511 views

How can I preprocess multi-page image inputs in a theano/lasagne network?

I am trying to classify multi-page documents using a convolutional neural network (CNN). The content of each page in the corpus contains only text (i.e., no photographs or icons), and different ...
jams's user avatar
  • 21
2 votes
1 answer
49 views

R: substituting only whole statements (not parts of them)

I have a data set DS. I want to substitute the statements contained with others following a pattern-replacement schema like the following one: I have used the followng function: ...
Andrea Ianni's user avatar
1 vote
1 answer
109 views

Using predictive modelling for temperature data set

I am absolutely new to this area of predictive modelling in data science. I am not able to understand how and what modelling techniques do we use? Does it depend on the data type? Does it depend on ...
Sagar Ghai's user avatar
2 votes
1 answer
120 views

Machine learning worker performance features for optimum allocation of tasks to workers

Suppose we have an English to French translation task in a company, and there are 100s of workers who are proficient in doing this task, but each worker has its own unique attributes which enables ...
data101's user avatar
  • 191
9 votes
7 answers
16k views

Python library that can compute the confusion matrix for multi-label classification

I'm looking for a Python library that can compute the confusion matrix for multi-label classification. FYI: scikit-learn doesn't support multi-label for confusion matrix) What is the difference ...
Franck Dernoncourt's user avatar
2 votes
1 answer
169 views

Theano logistic regression understanding code

I am used to C/Java like programming, and sometimes I am getting a headache on understanding the Python notation. On the logistic regression code available online, I am trying to understand this line ...
mlakhal's user avatar
  • 21
0 votes
1 answer
458 views

Multi-class logistic regression

I am trying to understand logistic regression for a multi-class example and I have the following code: ...
user's user avatar
  • 1,993
2 votes
3 answers
17k views

How to get common values between two multi sets?

I am working in R. I have two vectors; A and B of lengths 5913 and 3733 respectively (with entries are repeated). I want to extract those values (with repetitions) that are present in both A and B. I ...
user62198's user avatar
  • 1,091
3 votes
1 answer
946 views

How to determine Nonnegativity in Matrix Factorization?

We have information about what the user likes in our app and we want to recommend content to similar users even those who may not have explicitly like a particular content but are similar to those who ...
RAbraham's user avatar
  • 187
4 votes
2 answers
119 views

Web services to mine the social web?

Are there any web services that can be used to analyse data in social networks with respect to a specific research question (e.g. mentioning of certain products in social media discussions)?
orschiro's user avatar
  • 143
4 votes
2 answers
71k views

R: comparison (6) is possible only for atomic and list types

I am getting the error in the subject line for the following piece of code in R. ...
UD1989's user avatar
  • 258

15 30 50 per page
1
701 702
703
704 705
736