Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [databases]

A comprehensive collection of related data organized for convenient access, generally associated with software to update and query the data.

0
votes
0answers
4 views

How do I create a universal customer ID within Google Cloud Platform? [closed]

I had a few quick questions regarding the creation of a Universal Customer ID (UCID) within Google Cloud Platform (GCP). To make this easy, I'll start by providing some background, context, and ...
0
votes
1answer
17 views

Replacing column values in pandas with specific column with multiple database operation?

Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can ...
1
vote
0answers
10 views

Database System for Manual Entry

I'm in charge of setting up a patient register (100K+ patients) for a non-profit project with little money. This register should provide the basis for later datascience. I'm not sure how a good ...
2
votes
0answers
11 views

Database for storing nested s3 objects from broom, sweep, or tidy in R

I'm working on a product that requires automated forecasting of time-series data in R & Shiny. Using R's sweep, broom, & tidy packages, I get these 'tibbles', which are like dataframes with ...
0
votes
2answers
39 views

Database on time to churn in telecomunication

I would like to research about time to churn1 in the telecommunication market. Does anyone have a link to such a database? The only ones I found did not include the time of churn, but only if a ...
1
vote
0answers
13 views

Local U.S. City Government Expenditures APIs?

I'm currently working on a project that will compile the various expenditures of local city governments (Health, Transportation, etc), what initiatives are present in those major cities, and more ...
0
votes
1answer
61 views

Trying to get Orange connected to Postgres Database

I recently came across Orange, and am very keen to try it for our company's purposes (around 20 potential users, perhaps more). I have installed Orange (latest version) and am trying to connect to our ...
0
votes
0answers
37 views

Should data scientists be able to model a database schema or create ER diagrams?

I've started a university course in data science and the assigment set for learning SQL involves modelling a database schema - I thought this role was assigned to data architects/database ...
2
votes
1answer
48 views

Big Data - Data Warehouse Solutions?

I have a dozen of databases that stores different data, and each of them are 100TBs in size. All of the data is stored in AWS services such as RDS, Aurora and Dynamo. Many times I find myself need to ...
1
vote
0answers
1k views

Not enough replicas available for query at consistency all (2 required but only 1 alive)

I have a Cassandra cluster of 2 nodes and a keyspace "test" with a replication factor of 2. The consistency level is All. I am running a spark job which dumps data in Cassandra database. So, the ...
0
votes
0answers
38 views

How to transform data sets into data sequences

I'm studying sequence mining algorithms and for that, I read that I should use a transaction/sequence database just like this: <...
1
vote
1answer
775 views

Is there an overview over recommender system architectures?

I want to learn more about the recommender system topic. I am very interested in the usage of different database systems for this use case. My problem is that I cannot find a good overview over ...
1
vote
1answer
293 views

Are there databases specializing in scientific data

I am currently comparing file formats (HDF5 , etc.) to DBMS systems for a scientific data repository. I know of proprietary solutions such as Oracle extensions, but are there open source/free systems ...
1
vote
0answers
368 views

2D Image-based Avatar Animation With Expressions / Image-To-Image Translation

What I want to do Input an image-sequence (videoframes) that is already cropped and aligned to only the face/hair region and photorealistically replace this Person, Person A (can be anyone, I also ...
2
votes
1answer
32 views

MS Access issue [closed]

I have a database that I send to an excel sheet that auto populates some field in a report with calculations that I need. There are several users and the excel sheet cannot be open at multiple times ...
5
votes
3answers
4k views

Advantages of pandas dataframe to regular relational database

In Data Science, many seem to be using pandas dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to regular relational databases like MySQL, which ...
11
votes
2answers
2k views

Best practices to store Python machine learning models

What are the best practices to save, store, and share machine learning models? In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ...
5
votes
2answers
109 views

Seeking advice on database architecture — given my problem, what tools should I learn?

I'm a fairly experienced R user, but until now I haven't had a good reason to learn to use databases. Now I have a problem where I am dealing with model output that I need to save to disk, and then ...
-1
votes
1answer
19 views

How to document operations on a database? [closed]

I am working on a scientific article, where I am going to work with a large SQL based database. This work consist of everything from washing data, creating sub-databases etc. How can I document this ...
1
vote
2answers
592 views

Does SQL Server support the Poisson distribution?

I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson ...
1
vote
1answer
52 views

Is there a term for “this month last year” in a report? [closed]

I'm building a report that has month over month data, but also "this month last year". Is there a better/standard way of describing this?
1
vote
3answers
755 views

Instead of one-hot encoding, can I store the same information in one column using a single value?

Rather than creating 15 additional columns full of sparse binary data, could I: 1) use the first 15 prime numbers as indexes for the 15 categories 2) store data by multiplying the prime numbers of ...
2
votes
2answers
3k views

Classified Enron Email Dataset

I'm building a system able to classify emails into different categories (positive, negative, out of office, etc...) and I'm looking for a dataset of already classified emails to avoid hand ...
-1
votes
1answer
164 views

is there big difference between data Science , big Data and database? [closed]

is there big difference between data Science , big Data and database? i am confused in these three can anyone help me to out of this confusion?
1
vote
0answers
54 views

Analytics oriented database

I'm analyzing a single large table (about 50GB 16M rows, 15 columns perhaps more) currently using PostgreSQL 9.4. Most of my queries doesn't involve more tables. Queries run time are very large. I'm ...
2
votes
1answer
712 views

word2vec -storing a word and its vector yet be able to efficiently run k-nearest neighbour

After training a model using word2vec I'd now like to store the trained model with the word serving as a key and the vector as its value. However I'm not sure how I'll be able to implement in this way ...
0
votes
3answers
1k views

@RISK Vs R. Monte Carlo simulation in R?

Are there any cheaper or open source alternatives to @RISK or are there packages for R that would be able to perform the same tasks? http://www.palisade.com/risk/ @RISK (pronounced “at risk”) ...
1
vote
1answer
560 views

Implementing Frequently bought together using a DB

We have a classic structure of an online shop database (products, customers, sales) and we want to implement a Frequently bought together feature. Our software is in ASP.NET and we do not know PHP to ...
1
vote
1answer
430 views

Overcome memory limitation when downloading from database into Orange

I would like to run the association rule mining algorithm of the Orange library on a dataset that is stored in a PostgreSQL database. The table 'buildingset' contains the itemsets for each user, thus ...
0
votes
0answers
159 views

Retrieve data from Azure documentDB to R

I would like to retrieve data from an azure DocumentDB to be able to process them with R. I'm not able to find a package to do it. Have someone an idea on how to proceed?
2
votes
1answer
227 views

NoSQL vs SQL backend for semi structured data

I have a corpus of job descriptions and another corpus of CVs of applicants. I plan to implement a matching system using machine learning algorithms, to find top 5 or top 10 applicants for each job ...
0
votes
0answers
282 views

Merging Two Tables

I have FFT frequency feed from a particular Node . To make the time difference linear , I have a day into four group(6 hours each) and now i would like to insert the missing rows . For example see ...
2
votes
0answers
1k views

Decode SQLite database blobs: how to start?

Updated: see bottom of this question. Not sure if I am at the right channel here but I have good hopes someone here might be able to help me. I am trying to process and analyse data from a Footscan ...
2
votes
2answers
241 views

Data structure design for supporting arbitrary number of columns in table or database

I'm currently working on a sort-of a meta-modeler to build a free web service so that people can input data and run several models on that data. The task I'm currently struggling is: user needs to ...
20
votes
4answers
7k views

What makes columnar databases suitable for data science?

What are some of the advantages of columnar data-stores which make them more suitable for data science and analytics?
0
votes
1answer
46 views

Looking for a 'CITY, STATE' within a body of text (from a CITY-STATE database)

I'm looking for an optimal way to search a large body of text for a combination of words that resemble any CITY, STATE combination I have in a separate CITY-STATE database. My only idea would be to ...
2
votes
3answers
143 views

Can Hadoop be beneficial when data is in database tables and not in a file system

I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured ...
1
vote
2answers
44 views

Merging two datasets with unclear identifier in Excel

I'm trying to merge two datasets on Mergers & Acquisitions. They both consist of c.10'000 observations with c.50-100 variables each. One contains information about the actual M&A deal whereas ...
1
vote
2answers
703 views

any reason for this project to use hadoop/spark?

I am setting up for a couple self study projects to explore machine learning techniques. 1st project has 10,000 time series with 24 float data points each day for 10 years (876 million points). I ...
4
votes
3answers
570 views

Storing Sensor Data for Analysis of the Office

I have currently been tasked with designing an application that tracks several different measurements around the office, eg. the temperature, light, presence of people, etc. Having never really worked ...
2
votes
2answers
119 views

Data repositories like UCI

Are there any other data repositories like UCI and mlData, for biological data?? I want to know about mostly biological data set.
10
votes
3answers
6k views

Which is faster: PostgreSQL vs MongoDB on large JSON datasets?

I have a large dataset with 9m JSON objects at ~300 bytes each. They are posts from a link aggregator: basically links (a URL, title and author id) and comments (text and author ID) + metadata. They ...
1
vote
2answers
135 views

How to make an effective sampling from a database of text documents?

Problem: I want to know methods to perform an effective sampling from a database. The size of the database is about 250K text documents and in this case each text ...
38
votes
8answers
12k views

How to deal with version control of large amounts of (binary) data

I am a PhD student of Geophysics and work with large amounts of image data (hundreds of GB, tens of thousands of files). I know svn and ...
13
votes
1answer
7k views

Neo4j vs OrientDB vs Titan

I am working on a data-science project related on social relationship mining and need to store data in some graph databases. Initially I chose Neo4j as the database. But it seams Neo4j doesn't scale ...
1
vote
0answers
671 views

R-Cassandra connection with RJDBC

I am trying to connect with cassandra in R using RJDBC. When I execute casscon <- dbConnect(cassdrv, "jdbc:cassandra://ipaddrs:9160/demodb") I am getting <...
5
votes
2answers
3k views

Python interface to Titan Database

How can I connect to Titan database from Python ? What I understand is that Titan (Graph database) provides an interface (Blueprint) to Cassandra (Column Store) and bulb is a python interface to ...
2
votes
1answer
1k views

Database for a trie, or other appropriate structure for recommendation engine

We are storing the information about our users showing interest in our items. Based on this information, we would like to create a simple recommendation engine that will take the items I1, I2, I3 etc ...
4
votes
1answer
149 views

is this a good case for NOSQL?

I'm currently facing a project that I could solve with a relational database in a relatively painful way. Having heard so much about NOSQL, I'm wondering if there is not a more appropriate way of ...
12
votes
3answers
2k views

Efficient database model for storing data indexed by n-grams

I'm working on an application which requires creating a very large database of n-grams that exist in a large text corpus. I need three efficient operation types: Lookup and insertion indexed by the n-...