Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
2
votes
2answers
6k views

What is the difference between data-driven methods and machine learning?

I was wondering (about a more semantic question), is there a difference between data-driven methods and machine learning? Or is it more correct to state that machine learning is a category of data-...
0
votes
2answers
713 views

Connecting Infusionsoft data to Google data studio

I want to create a Google Data studio dashboard from Infusion soft data. The main problem are the connectors - there are multiple tools that provide direct connectors but they are paid solutions like ...
2
votes
1answer
139 views

How is a single element of the training set called?

This question is only about the vocabulary. Do / can you say data item data sample recording sample data point something else when you talk about elements of the training / test set? For example: ...
0
votes
2answers
345 views

How synthetic sampling of data helps classifier to predict better result?

I am currently working on data imbalance using SMOTE for binary and other algorithms for the multi-class problem. I have the idea how to create the synthetic example to bring noticeable accuracy on a ...
1
vote
3answers
1k views

Instead of one-hot encoding, can I store the same information in one column using a single value?

Rather than creating 15 additional columns full of sparse binary data, could I: 1) use the first 15 prime numbers as indexes for the 15 categories 2) store data by multiplying the prime numbers of ...
1
vote
2answers
892 views

What are recommended ways\tools for processing large data from Excel Files?

A Very Happy New Year! I'm currently working on an analytics project with large volumes of data stored in excel files (about 50GB in 1000 files). The files use a custom formatting to store date-time ...
0
votes
1answer
66 views

Measuring Difference Between Two Sets of Likert Values

I asked users to complete a likert survey (1-5) at the start of an activity, and conclusion of the activity. What is the best manner to show the rate of change/difference between these two result sets?...
2
votes
1answer
7k views

What is “noise” in observed data?

I am reading pattern Recognition and machine learning by Bishop and in the chapter about probability, "noise in the observed data" is mentioned many times. I have read on the internet that noise ...
2
votes
1answer
253 views

The best way to calculate variations between 2 datasets?

I am trying to write an application which identifies SQL query patterns. The program will trace all the queries executed in a database. When it identifies a considerable pattern change in the database,...
7
votes
1answer
9k views

How to generate training data for OCR

I am trying to build and optical character recognition system for recognizing license plate (Indonesian licence plat), unfortunately there is no training set available but I found the font, I try to ...
0
votes
1answer
52 views

Built a Heap in Cluster algorithm

Does anyone could explain in details how heap structure works in Cluster Algorithm? I am planning to code CURE in Matlab (Clustering using representatives) but, at first, the paper mentions that all ...
2
votes
2answers
108 views

How many people can use a single Hadoop cluster at one time?

I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a ...
0
votes
1answer
59 views

Estimating forward velocity for a swimmer

With a modern IMU with 9 angles of freedom collecting accelerometer, magnetometer and gyroscope data on 3 axis, what would be the best approach on filtering the data and handling it to accurately ...
0
votes
1answer
112 views

How do I write a SQL (or dwSQL) query across these two different datasets?

I'm having a little fun and I'd love to know how to write a SQL or dwSQL query across these two datasets of Trumps tweets and Facebook posts. Here are the datasets: Trump Facebook Posts https://data....
0
votes
1answer
25 views

How not to see the output in the console when adding a new column in a table?

I have a data table called td and it contains 14k obs and 5k variables. When I run the command td[,fv_mc2:=1] I also get the whole data table as output in my ...
1
vote
0answers
68 views

Econometrics Thesis methodology Suggestions?

I just picked my topic for my econometrics course thesis. I decided to create an algorithm that forecasts NBA players' performance. My inspiration is Silver Nate's PECOTA and CARMELO algorithms, but I ...
1
vote
3answers
2k views

Forecasting sales of next year using sales of past years?

I have a dataset of sales of a company for different products throughout 3 years. I have to forecast the sales for each of these products for next year. A sample of the dataset is: Here I have to ...
1
vote
1answer
66 views

Postitive event but unsure when it occurred in time

Morning, I have a lot of data of which I am positive an event (target data / the event I want to predict going forward) occurred within a two week time frame, however I am unsure when it happened ...
0
votes
1answer
472 views

When to apply binning?

I have read quite a lot about discretization techinques, measuring WoE and IV etc., but the basic question - when is it worth to use binning and when isn't - is not entirely clear. On the one hand, ...
2
votes
1answer
302 views

Directly proportional Trend in training and cross validation curves

In continuation with a question already asked, I tried the same curve on a different dataset I found. My model is a simple Logistic regression curve with OnevsRest Classifier. But the graph I obtained ...
2
votes
2answers
2k views

Dealing with outliers and z-scores

I am new to Data Science and have a few trivial questions, which I think are essential for my understanding of the basic data science techniques. I am building a function to calculate a social ...
2
votes
0answers
129 views

How to quantitatively compare two or more complex data sets

I have a series of datasets that are composed of 100 or so variables and a corresponding response variable. I am often faced with the question of trying to attribute differences in the response ...
0
votes
2answers
74 views

How should one deal with null values in time stamp data types? [closed]

I have a data set where certain features are time stamps for certain important events like time stamp of when a user logged in etc. How should I deal with null values in such cases. Should one just ...
1
vote
1answer
6k views

Target variable problem :- Classifier [closed]

I installed orange version 3 yesterday. I defined variables in excel file as per instruction given in documentation. When I used simple plots it works fine but in case of classifier it's not working. ...
4
votes
2answers
577 views

What is the most used format to save data with type information

I am exporting data from an SQL database and importing it into R. This is a two step process since I first (automatically) download the data to a hard drive and then import the file with R. ...
4
votes
8answers
562 views

Skills that school doesn't teach you

I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my ...
2
votes
2answers
850 views

Multi-touch Attribution Model

I would like to develop an algorithm for Multi-Touch Time-Decay Attribution Model for an e-mail campaign. I have the research paper ready and I would convert the paper into R-script. I am struggling ...
2
votes
1answer
42 views

Difficulties of getting raw data

I am trying to obtain raw data for (violent) crime rates of a US/Canadian city (any city would do), but I need the data to be granular and raw. All I could find is either interpretations, summary data ...
3
votes
2answers
12k views

Merging large CSV files in Pandas

I have two CSV files(each of the file size is in GBs). I am trying to merge the two CSV files but every time I try to it my computer hangs. Is there no way to merge the files in chunks in pandas ...
1
vote
4answers
1k views

Which one is better performer on wrangling big data, R or Python?

I want to know which language/packages performs better & faster on wrangling big data? R and Python both has packages and libraries for wrangling and cleaning data. But which packages and ...
7
votes
4answers
377 views

How to handle the CEO expectations from a company that's new to data science?

I'm new here. I'm about to have a final interview for a data scientist position for a company (it's in the e-commerce field) that is new for data science. It's a pretty new position for the company, ...
2
votes
3answers
109 views

Data sources problem

I’m doing an “analytics” project in which I have to collect data from small businesses (SMEs) in South America and try to come up with some insights about what are their needs in general. The project’...
-3
votes
1answer
1k views

I have 11 years experience on .net, should i learn data science? [closed]

I have 11 years experience on .net, should i learn data science? I have 11 years experience on .net and getting paid rs 14L in india noida, Should my experience get consider in Data Science and get ...
1
vote
2answers
104 views

First steps when analyzing a company's data

I'm not sure if this question is appropriate for this forum, so excuse me if it's not (if not, any suggestions on where might be a better place would be very much appreciated). I'm currently an ...
0
votes
1answer
167 views

Verification of trained system

I have trained a system in order to detect some features from a set of scenarios. Now the system can detect and classify that set. How can I validate how that system works in real world? What ...
0
votes
1answer
40 views

Daily access pattern of a real cloud storage

for a current research project I would need file access traces of real world cloud storages. Unfortunately I was not able to find any yet. More precisely I need real world traces with at least the ...
1
vote
3answers
858 views

Where can I get a comprehensive criminal dataset?

I want to create a machine learning model to predict the probability of Person committing crime in future. Given his/her past crime history, age, gender, race, employment history, family information (...
0
votes
3answers
3k views

@RISK Vs R. Monte Carlo simulation in R?

Are there any cheaper or open source alternatives to @RISK or are there packages for R that would be able to perform the same tasks? http://www.palisade.com/risk/ @RISK (pronounced “at risk”) ...
2
votes
0answers
151 views

How can I find company descriptions for a long list of companies?

I'm going to train an ml algorithm to qualify potential sales leads based upon company descriptions. To do this, I need to find the company descriptions programatically. E.g. given a long list of ...
2
votes
3answers
2k views

Finding outliers in multiple dimensions

I'm working on a dataset which isn't normally distributed. The dataset contains three dimensions like cost, discount and profit. I'm trying to find possible outliers in all these dimensions. I used ...
3
votes
4answers
99 views

About data cleansing, to what extent should we do our work?

As far as I know, when we deal with 'big data'. It is common we deal with more than 10 years' customers data. Quality issue is always there. What I am thinking about, to what extend we should deal ...
3
votes
3answers
312 views

Algorithm or formula to measure happiness?

I am working on a project that takes in comments in a given Facebook page and determines the average happiness rating for it based on users of the page. My question is where would I find a formula, ...
2
votes
2answers
267 views

Retail Store Testing

I am an analyst who is new to retail store testing (for example: customer response to a store new layout or a new product or process in one or 2 stores). Any suggestions for a good book/blog/any ...
0
votes
4answers
187 views

A new comer in this field - How to start?

To be honest I find entire DS field so vague that I can't find actual roadmap of learning. I am actually a programmer for 10+ years and pretty much interested to get into ...
5
votes
4answers
4k views

Why aren't languages like C, C++ used for data analytics instead of R, Python?

I have started learning data science using R, however I have C++ as a subject this semester, and my project is to predict the outcome of a game using C++. I have not come across many instances (close ...
2
votes
0answers
60 views

Core components of data literacy for working professionals

I am a social scientist invited to give a 90-minute seminar on data literacy for working professionals in the U.S. government. I've struggled to develop a concise definition of data literacy, but ...
1
vote
3answers
75 views

Creating queries on the fly and general manipulation for dataset of half a million data records

What would one use for manipulating data of the kind below ? a) Data is bio-markers of different globs GlobA 3 4 5 .... GlobB 2 1 1 .... GlobC 3 2 1 .... b) Manipulations are queries like: - show ...
0
votes
1answer
22 views

How to Ascertain Sine Wave and Harmonics

I have some data that I've noticed conforms to a sine wave. Details of the data is unknown. My task is to approximate it as closely as possible. From some experimentation in Excel, I noticed the data ...
2
votes
0answers
52 views

Free/open interactive softwares/plugins for end-users' high-dimensional data visualization

Aside from questions about How to visualize data of a multidimensional dataset (TIMIT), the Purpose of visualizing high dimensional data? and High-dimensional data: What are useful techniques to know?,...
0
votes
1answer
51 views

Add new factor across multiple groups

I am trying to add multiple new rows for 3 new factor level in an existing data frame. Please refer to sample data for an example. My starting data frame has 18 levels for ...

1 6 7 8 9 10