Questions tagged [data]
Questions mostly concerned with managing data, without focus on pre-processing or modelling.
863
questions
0
votes
2
answers
95
views
How should one deal with null values in time stamp data types? [closed]
I have a data set where certain features are time stamps for certain important events like time stamp of when a user logged in etc. How should I deal with null values in such cases. Should one just ...
1
vote
1
answer
9k
views
Target variable problem :- Classifier [closed]
I installed orange version 3 yesterday. I defined variables in excel file as per instruction given in documentation. When I used simple plots it works fine but in case of classifier it's not working. ...
4
votes
2
answers
1k
views
What is the most used format to save data with type information
I am exporting data from an SQL database and importing it into R. This is a two step process since I first (automatically) download the data to a hard drive and then import the file with R.
...
- 961
5
votes
8
answers
645
views
Skills that school doesn't teach you
I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my ...
- 51
2
votes
2
answers
931
views
Multi-touch Attribution Model
I would like to develop an algorithm for Multi-Touch Time-Decay Attribution Model for an e-mail campaign. I have the research paper ready and I would convert the paper into R-script. I am struggling ...
- 21
2
votes
1
answer
71
views
Difficulties of getting raw data
I am trying to obtain raw data for (violent) crime rates of a US/Canadian city (any city would do), but I need the data to be granular and raw. All I could find is either interpretations, summary data ...
- 121
7
votes
2
answers
23k
views
Merging large CSV files in pandas
I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
- 3,001
1
vote
4
answers
3k
views
Which one is better performer on wrangling big data, R or Python?
I want to know which language/packages performs better & faster on wrangling big data? R and Python both has packages and libraries for wrangling and cleaning data. But which packages and ...
7
votes
4
answers
452
views
How to handle the CEO expectations from a company that's new to data science?
I'm new here. I'm about to have a final interview for a data scientist position for a company (it's in the e-commerce field) that is new for data science. It's a pretty new position for the company, ...
- 73
2
votes
3
answers
122
views
Data sources problem
I’m doing an “analytics” project in which I have to collect data from small businesses (SMEs) in South America and try to come up with some insights about what are their needs in general. The project’...
- 163
-3
votes
1
answer
2k
views
I have 11 years experience on .net, should i learn data science? [closed]
I have 11 years experience on .net, should i learn data science?
I have 11 years experience on .net and getting paid rs 14L in india noida,
Should my experience get consider in Data Science and get ...
1
vote
2
answers
110
views
First steps when analyzing a company's data
I'm not sure if this question is appropriate for this forum, so excuse me if it's not (if not, any suggestions on where might be a better place would be very much appreciated). I'm currently an ...
- 13
0
votes
1
answer
177
views
Verification of trained system
I have trained a system in order to detect some features from a set of scenarios. Now the system can detect and classify that set. How can I validate how that system works in real world? What ...
- 103
0
votes
1
answer
43
views
Daily access pattern of a real cloud storage
for a current research project I would need file access traces of real world cloud storages. Unfortunately I was not able to find any yet.
More precisely I need real world traces with at least the ...
- 1
1
vote
3
answers
1k
views
Where can I get a comprehensive criminal dataset?
I want to create a machine learning model to predict the probability of Person committing crime in future. Given his/her past crime history, age, gender, race, employment history, family information (...
- 257
1
vote
3
answers
4k
views
@RISK Vs R. Monte Carlo simulation in R?
Are there any cheaper or open source alternatives to @RISK or are there packages for R that would be able to perform the same tasks?
http://www.palisade.com/risk/
@RISK (pronounced “at risk”) ...
- 21
2
votes
0
answers
362
views
How can I find company descriptions for a long list of companies?
I'm going to train an ml algorithm to qualify potential sales leads based upon company descriptions. To do this, I need to find the company descriptions programatically.
E.g. given a long list of ...
- 21
1
vote
1
answer
319
views
Association between Categorical Variables and regression
We perform data analysis and build models. Say, for example, I built a regression model that has more than one predictor (multiple regression). We then check many things: normality, multicollinearity, ...
2
votes
3
answers
2k
views
Finding outliers in multiple dimensions
I'm working on a dataset which isn't normally distributed. The dataset contains three dimensions like cost, discount and profit.
I'm trying to find possible outliers in all these dimensions. I used ...
- 151
3
votes
4
answers
112
views
About data cleansing, to what extent should we do our work?
As far as I know, when we deal with 'big data'. It is common we deal with more than 10 years' customers data. Quality issue is always there.
What I am thinking about, to what extend we should deal ...
- 171
3
votes
3
answers
415
views
Algorithm or formula to measure happiness?
I am working on a project that takes in comments in a given Facebook page and determines the average happiness rating for it based on users of the page. My question is where would I find a formula, ...
- 133
2
votes
2
answers
340
views
Retail Store Testing
I am an analyst who is new to retail store testing (for example: customer response to a store new layout or a new product or process in one or 2 stores).
Any suggestions for a good book/blog/any ...
- 131
0
votes
4
answers
204
views
A new comer in this field - How to start?
To be honest I find entire DS field so vague that I can't find actual roadmap of learning. I am actually a programmer for 10+ years and pretty much interested to get into ...
- 341
6
votes
4
answers
11k
views
Why aren't languages like C, C++ used for data analytics instead of R, Python?
I have started learning data science using R, however I have C++ as a subject this semester, and my project is to predict the outcome of a game using C++.
I have not come across many instances (close ...
2
votes
0
answers
98
views
Core components of data literacy for working professionals
I am a social scientist invited to give a 90-minute seminar on data literacy for working professionals in the U.S. government.
I've struggled to develop a concise definition of data literacy, but ...
- 21
1
vote
3
answers
78
views
Creating queries on the fly and general manipulation for dataset of half a million data records
What would one use for manipulating data of the kind below ?
a) Data is bio-markers of different globs
GlobA 3 4 5 ....
GlobB 2 1 1 ....
GlobC 3 2 1 ....
b) Manipulations are queries like:
- show ...
- 13
0
votes
1
answer
59
views
How to Ascertain Sine Wave and Harmonics
I have some data that I've noticed conforms to a sine wave. Details of the data is unknown. My task is to approximate it as closely as possible.
From some experimentation in Excel, I noticed the data ...
- 398
2
votes
0
answers
63
views
Free/open interactive softwares/plugins for end-users' high-dimensional data visualization
Aside from questions about How to visualize data of a multidimensional dataset (TIMIT), the Purpose of visualizing high dimensional data? and High-dimensional data: What are useful techniques to know?,...
- 1,447
0
votes
1
answer
103
views
Add new factor across multiple groups
I am trying to add multiple new rows for 3 new factor level in an existing data frame. Please refer to sample data for an example. My starting data frame has 18 levels for ...
- 427
7
votes
2
answers
2k
views
IID violation in machine learning
Imagine I'm collecting some training data. Lets say I collect a 30minute time series from 1000 people so I have 1000 observations (rows) in my feature matrix. I train some model (lets say a neural ...
- 1,071
5
votes
3
answers
4k
views
What are the methods to ensure that the population split for A/B test is random?
Before launching an A/B test, what are the methods to ensure that the population split in control and target group is random for a particular label say, purchase rate.
- 91
5
votes
4
answers
478
views
General way to reduce features
Let's say I have a giant dataset (600+ columns) and I have no idea what insights I might get from it or what model I want to run.
What are some of the best ways to find the most influential columns/...
2
votes
0
answers
71
views
Data Scientists | Sectors , Departments and Profiles [closed]
I'm searching a lot about what people call Data Scientist. From my brief search, I've came with the conclusion that Data Scientists' are probably what you may call the future evolution of a Business ...
- 121
2
votes
0
answers
67
views
Tag categorizer
I'm using Google prediction API and I need to perform tag categorization for free text. So when I receive a text like "Cant update application" is tagged with #Apps #General #CantUpdate.
My ...
- 749
2
votes
1
answer
712
views
PCA on acceleration time series data
Has anyone attempted Principal component analysis on time series and acceleration data(ie. data from accelerometer and sensors) and tried compressing it as well as regenerating back the data with ...
- 21
2
votes
2
answers
398
views
Data structure design for supporting arbitrary number of columns in table or database
I'm currently working on a sort-of a meta-modeler to build a free web service so that people can input data and run several models on that data.
The task I'm currently struggling is: user needs to ...
-2
votes
1
answer
2k
views
create a company to make money by winning Kaggle competition [closed]
I am thinking of creating a data science company (team) that aims to make money by winning Kaggle competitions. It will be actually a onsite data science learning service but I want to create the ...
- 101
3
votes
4
answers
5k
views
Mathematics major for data science
So I'm a recent transfer 2nd year student from Computer Science major to Mathematics major. Though I do have a bit of an issue here. I can choose between the applied mathematics, pure mathematics and ...
- 33
13
votes
3
answers
34k
views
How to create US state choropleth map
I have a value associated with each US state (let's pretend it's the average temperature in January for each state). I want to display this data as a heat map of the United States. To be clear, it ...
- 131
4
votes
1
answer
56
views
How to deal with analyzing optional survey data
Consider this scenario. There is a website that ranks every type of sandwich. For each type of sandwich, users fill out a survey about the following: A) how tasty it is (on a scale from 1-5); B) how ...
- 41
3
votes
0
answers
666
views
Central Probability Interval
So, I'm having problems learning for a data analysis lecture. The topic is the construction of the "central interval" as our lecturer calls it. The concept shall path the way to understand confidence &...
- 31
13
votes
1
answer
4k
views
Do I have to standardize my new polynomial features?
I have a vector X with n features previously standardized.
If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
- 619
2
votes
0
answers
56
views
How to select a bunch of optimized data from a larger data set?
How to select a bunch of optimized data from a larger data set? I have a data set containing the products to be sold to the customers. And the products were sold before, it has some information, like ...
- 141
1
vote
4
answers
183
views
What kind of research I can do with my data set?
I have a data containing 200K ads (sale and rent property in the United Arab Emirates, or UAE). Here is a sample:
...
7
votes
4
answers
3k
views
Tool to Generate 2D Data via Mouse Clicking
Often when I am learning new machine learning methods or experimenting with a data analysis algorithm I need to generate a series of 2D points. Teachers also do this often when making a lesson or ...
- 280
0
votes
1
answer
82
views
how often should one sample a dataset?
I have some archive data that show the variation of a certain quantity X measured over 1-year intervals.
I'm now facing the problem of understanding how X correlates with another variable Y, that I ...
- 143
3
votes
1
answer
66
views
Establishing data science programs as an independent discipline
What are the benefits of establishing data science programs as an independent discipline rather than creating data scientists in different fields? (for students, universities and the society as a ...
- 910
4
votes
3
answers
1k
views
Deploying machine learning modules
I am looking to find some resources about what I wan't to do :I wan't to make some GUI of my machine learning models and finally deploy them as a web app.I find R Shiny to be somehow ok , but it ...
- 133
1
vote
0
answers
48
views
Generate a set of abstract search terms
I am developing an application for research purposes that uses randomly generated tweets to detect emerging conceptual topics. I need to collect a large amount of them (let's say, beyond 5 million) ...
- 111
3
votes
2
answers
207
views
Domain-specific data science programs
I am looking for domain-specific data science programs (as a major, not a minor or specialization). I found programs in bio-statistics (and other disciplines in public health), MS in marketing ...
- 910