Questions tagged [databases]

A comprehensive collection of related data organized for convenient access, generally associated with software to update and query the data.

Filter by
Sorted by
Tagged with
48
votes
9answers
21k views

How to deal with version control of large amounts of (binary) data

I am a PhD student of Geophysics and work with large amounts of image data (hundreds of GB, tens of thousands of files). I know svn and ...
30
votes
4answers
16k views

Best practices to store Python machine learning models

What are the best practices to save, store, and share machine learning models? In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ...
25
votes
4answers
9k views

What makes columnar databases suitable for data science?

What are some of the advantages of columnar data-stores which make them more suitable for data science and analytics?
13
votes
5answers
13k views

Advantages of pandas dataframe to regular relational database

In Data Science, many seem to be using pandas dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to regular relational databases like MySQL, which ...
13
votes
1answer
216 views

When a relational database has better performance than a no relational

When a relational database, like MySQL, has better performance than a no relational, like MongoDB? I saw a question on Quora other day, about why Quora still uses MySQL as their backend, and that ...
13
votes
1answer
7k views

Neo4j vs OrientDB vs Titan

I am working on a data-science project related on social relationship mining and need to store data in some graph databases. Initially I chose Neo4j as the database. But it seams Neo4j doesn't scale ...
12
votes
3answers
3k views

Efficient database model for storing data indexed by n-grams

I'm working on an application which requires creating a very large database of n-grams that exist in a large text corpus. I need three efficient operation types: Lookup and insertion indexed by the n-...
10
votes
2answers
919 views

Is this Neo4j comparison to RDBMS execution time correct?

Background: Following is from the book Graph Databases, which covers a performance test mentioned in the book Neo4j in Action: Relationships in a graph naturally form paths. Querying, or ...
10
votes
3answers
7k views

Which is faster: PostgreSQL vs MongoDB on large JSON datasets?

I have a large dataset with 9m JSON objects at ~300 bytes each. They are posts from a link aggregator: basically links (a URL, title and author id) and comments (text and author ID) + metadata. They ...
9
votes
3answers
4k views

Human activity recognition using smartphone data set problem

I'm new to this community and hopefully my question will well fit in here. As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using ...
9
votes
4answers
2k views

What is the ideal database that allows fast cosine distance?

I'm currently trying to store many feature vectors in a database so that, upon request, I can compare an incoming feature vector against many other (if not all) stored in the db. I would need to ...
6
votes
1answer
531 views

Is Data Science just a trend or is a long term concept? [closed]

I see a lot of courses in Data Science emerging in the last 2 years. Even big universities like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it looks like data ...
6
votes
2answers
122 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
6
votes
2answers
3k views

Python interface to Titan Database

How can I connect to Titan database from Python ? What I understand is that Titan (Graph database) provides an interface (Blueprint) to Cassandra (Column Store) and bulb is a python interface to ...
5
votes
2answers
626 views

Running an R script programmatically [closed]

I have an R script that generates a report based on the current contents of a database. This database is constantly in flux with records being added/deleted many times each day. How can I ask my ...
5
votes
1answer
164 views

is this a good case for NOSQL?

I'm currently facing a project that I could solve with a relational database in a relatively painful way. Having heard so much about NOSQL, I'm wondering if there is not a more appropriate way of ...
5
votes
3answers
845 views

Storing Sensor Data for Analysis of the Office

I have currently been tasked with designing an application that tracks several different measurements around the office, eg. the temperature, light, presence of people, etc. Having never really worked ...
4
votes
2answers
122 views

Seeking advice on database architecture — given my problem, what tools should I learn?

I'm a fairly experienced R user, but until now I haven't had a good reason to learn to use databases. Now I have a problem where I am dealing with model output that I need to save to disk, and then ...
4
votes
1answer
2k views

What is the difference between ImageNet and ImageNet1k? How to download it?

Some papers mention just ImageNet and some papers mention ImageNet 1k database? What is the difference between these 2? Are they same or is the latter one subset of the former one? I'm working on ...
4
votes
1answer
2k views

Database for a trie, or other appropriate structure for recommendation engine

We are storing the information about our users showing interest in our items. Based on this information, we would like to create a simple recommendation engine that will take the items I1, I2, I3 etc ...
3
votes
1answer
1k views

Is there an overview over recommender system architectures?

I want to learn more about the recommender system topic. I am very interested in the usage of different database systems for this use case. My problem is that I cannot find a good overview over ...
3
votes
1answer
171 views

Big Data - Data Warehouse Solutions?

I have a dozen of databases that stores different data, and each of them are 100TBs in size. All of the data is stored in AWS services such as RDS, Aurora and Dynamo. Many times I find myself need to ...
3
votes
3answers
387 views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
3
votes
1answer
50 views

How to find the values that never matched in a full join

Hello I have done this full join query: ...
3
votes
2answers
2k views

Data set with multiple tables

I want to check an algorithm that learns the connections between different tables in a database. Where can I find datasets that are compound of multiple tables (i.e. more than 5) that have ...
2
votes
3answers
157 views

Can Hadoop be beneficial when data is in database tables and not in a file system

I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured ...
2
votes
1answer
448 views

Are there databases specializing in scientific data

I am currently comparing file formats (HDF5 , etc.) to DBMS systems for a scientific data repository. I know of proprietary solutions such as Oracle extensions, but are there open source/free systems ...
2
votes
2answers
148 views

Data repositories like UCI

Are there any other data repositories like UCI and mlData, for biological data?? I want to know about mostly biological data set.
2
votes
1answer
274 views

NoSQL vs SQL backend for semi structured data

I have a corpus of job descriptions and another corpus of CVs of applicants. I plan to implement a matching system using machine learning algorithms, to find top 5 or top 10 applicants for each job ...
2
votes
2answers
6k views

Classified Enron Email Dataset

I'm building a system able to classify emails into different categories (positive, negative, out of office, etc...) and I'm looking for a dataset of already classified emails to avoid hand ...
2
votes
1answer
912 views

word2vec -storing a word and its vector yet be able to efficiently run k-nearest neighbour

After training a model using word2vec I'd now like to store the trained model with the word serving as a key and the vector as its value. However I'm not sure how I'll be able to implement in this way ...
2
votes
2answers
329 views

Data structure design for supporting arbitrary number of columns in table or database

I'm currently working on a sort-of a meta-modeler to build a free web service so that people can input data and run several models on that data. The task I'm currently struggling is: user needs to ...
2
votes
1answer
33 views

What is the best practice for data folder structuring?

I work for a small data science consultancy firm and we are trying to standardize our project folder structure. We started from the cookiecutter structure which is a great base. However one of the ...
2
votes
1answer
36 views

MS Access issue [closed]

I have a database that I send to an excel sheet that auto populates some field in a report with calculations that I need. There are several users and the excel sheet cannot be open at multiple times ...
2
votes
2answers
183 views

How to make an effective sampling from a database of text documents?

Problem: I want to know methods to perform an effective sampling from a database. The size of the database is about 250K text documents and in this case each text ...
2
votes
1answer
14 views

DBMS or Software for privacy sensitive data

We have a dataset of very privacy sensitive people data and want to build a database with it. The data protection department in our company doesn't like the idea that the data scientists are able to ...
2
votes
0answers
579 views

2D Image-based Avatar Animation With Expressions / Image-To-Image Translation

What I want to do Input an image-sequence (videoframes) that is already cropped and aligned to only the face/hair region and photorealistically replace this Person, Person A (can be anyone, I also ...
2
votes
0answers
2k views

Decode SQLite database blobs: how to start?

Updated: see bottom of this question. Not sure if I am at the right channel here but I have good hopes someone here might be able to help me. I am trying to process and analyse data from a Footscan ...
2
votes
0answers
646 views

What are the advantages and disadvantages of SQL versus NoSQL in data science? [closed]

In which situations would one system be preferred over the other? What are the relative advantages and disadvantages of relational databases versus non-relational databases?
1
vote
3answers
1k views

Instead of one-hot encoding, can I store the same information in one column using a single value?

Rather than creating 15 additional columns full of sparse binary data, could I: 1) use the first 15 prime numbers as indexes for the 15 categories 2) store data by multiplying the prime numbers of ...
1
vote
2answers
777 views

any reason for this project to use hadoop/spark?

I am setting up for a couple self study projects to explore machine learning techniques. 1st project has 10,000 time series with 24 float data points each day for 10 years (876 million points). I ...
1
vote
1answer
860 views

How to plot time series data from SQLite database

I am trying to plot time series data by accessing the SQLite database. I successfully connect the database by coding the below script,however, i couldn't figure out how to plot the time series data ...
1
vote
3answers
160 views

Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?

It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model ...
1
vote
1answer
421 views

Trying to get Orange connected to Postgres Database

I recently came across Orange, and am very keen to try it for our company's purposes (around 20 potential users, perhaps more). I have installed Orange (latest version) and am trying to connect to our ...
1
vote
1answer
1k views

Implementing Frequently bought together using a DB

We have a classic structure of an online shop database (products, customers, sales) and we want to implement a Frequently bought together feature. Our software is in ASP.NET and we do not know PHP to ...
1
vote
1answer
108 views

Where to find height dataset, or datasets in General

Hi there smart people, I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest. For example, I ...
1
vote
2answers
178 views

Null values as useful information for feature engineering

I'm preparing features for a neural network which I'll run in Keras and TensorFlow. The features are generated in Oracle. There I also replace null values. I'm not doing normalization on the database ...
1
vote
1answer
232 views

Machine Learning applied to database design [closed]

Is it possible to apply machine learning to database design given a data source with (pseudo) relationships?
1
vote
2answers
1k views

Does SQL Server support the Poisson distribution?

I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson ...
1
vote
2answers
55 views

Merging two datasets with unclear identifier in Excel

I'm trying to merge two datasets on Mergers & Acquisitions. They both consist of c.10'000 observations with c.50-100 variables each. One contains information about the actual M&A deal whereas ...