Questions tagged [databases]

A comprehensive collection of related data organized for convenient access, generally associated with software to update and query the data.

Filter by
Sorted by
Tagged with
2
votes
1answer
17 views

DBMS or Software for privacy sensitive data

We have a dataset of very privacy sensitive people data and want to build a database with it. The data protection department in our company doesn't like the idea that the data scientists are able to ...
0
votes
0answers
5 views

How to store and query biometric data for an authentication system?

I am trying to design, and hopefully implement, an authentication system which centers around the use of biometric images. I plan to use different machine learning and deep learning techniques to help ...
0
votes
0answers
10 views
0
votes
0answers
15 views

Data storage for Intrusion Detection System

I write an IDS(in C) for a KNX network for my thesis. For this, I just store all the telegrams within an SQLite database. But I'm not sure if a SQL database is the best here. I want to "ask ...
0
votes
0answers
13 views

Manually rollbacking JDBC transaction or just leaving it to the database

When managing transactions through JDBC, should I manually rollback if some exception occurs or leave it to the database to do the rollback on its own? The database in question is Postgresql. As far ...
1
vote
2answers
173 views

ML model deployment architecture?

I came from a software development background and we have separate servers of the same database (dev, test, prod). The reason for this is because we develop our apps against the dev DB, run tests ...
1
vote
1answer
17 views

I Have Issues Installing Basemap

I tried to install Basemap and it gives me this: preparing transaction: done verifying transaction: done executing transaction: failed ERROR conda.core.link:_execute(507): An error occurred while ...
0
votes
1answer
19 views

Azure Cloud SQL - Querying large number of rows with Python

I have a Python Flask application that connects to an Azure Cloud SQL Database, and uses the Pandas read_sql method with SQLAlchemy to perform a select operation on a table and load it into a ...
9
votes
4answers
2k views

What is the ideal database that allows fast cosine distance?

I'm currently trying to store many feature vectors in a database so that, upon request, I can compare an incoming feature vector against many other (if not all) stored in the db. I would need to ...
3
votes
3answers
396 views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
3
votes
2answers
2k views

Data set with multiple tables

I want to check an algorithm that learns the connections between different tables in a database. Where can I find datasets that are compound of multiple tables (i.e. more than 5) that have ...
6
votes
2answers
122 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
9
votes
3answers
4k views

Human activity recognition using smartphone data set problem

I'm new to this community and hopefully my question will well fit in here. As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using ...
10
votes
3answers
7k views

Which is faster: PostgreSQL vs MongoDB on large JSON datasets?

I have a large dataset with 9m JSON objects at ~300 bytes each. They are posts from a link aggregator: basically links (a URL, title and author id) and comments (text and author ID) + metadata. They ...
0
votes
0answers
14 views

Machine Learning Recommender high data intensity

We are building a recommender engine to be integrated in an app that, each time an API is called, will pull thousands of records from an Azure SQL database and create recommendations. Currently with ...
30
votes
4answers
16k views

Best practices to store Python machine learning models

What are the best practices to save, store, and share machine learning models? In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ...
0
votes
0answers
76 views

Use Tableau with Graph Database Neo4j to Visualize Pickup and Dropoff Points in Different Colors

I'm tryin to visualize taxi trips in NYC with Tableau. I've stored the data in a graph database (Neo4J) and connected Tableau to the database via a JDBC connector. My problem is, that I want to ...
0
votes
0answers
31 views

Database augmentation using Sobel filter (aka gradient)

I'm working in PyTorch and to improve a ProGAN training I'm augmenting a database by calculating, for each image, the horizontal and vertical gradient via the Sobel operator. Do you have any reading ...
13
votes
5answers
13k views

Advantages of pandas dataframe to regular relational database

In Data Science, many seem to be using pandas dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to regular relational databases like MySQL, which ...
0
votes
0answers
17 views

Project structure ML time-series forecasting

I'm about to start with a ML learning time-series forecasting project which includes large-quantities of (2-million) time-series stored in JSON-files. The first step will involve feature engineering....
48
votes
9answers
21k views

How to deal with version control of large amounts of (binary) data

I am a PhD student of Geophysics and work with large amounts of image data (hundreds of GB, tens of thousands of files). I know svn and ...
1
vote
1answer
495 views

Connect Orange 3.20 to postgresql database

I installed orange 3.20 on windows 7. It works so far, the problem is connecting it to a server based Postgres database. While the connection can be made in the moment, when you try to load a table ...
1
vote
1answer
872 views

How to plot time series data from SQLite database

I am trying to plot time series data by accessing the SQLite database. I successfully connect the database by coding the below script,however, i couldn't figure out how to plot the time series data ...
2
votes
1answer
34 views

What is the best practice for data folder structuring?

I work for a small data science consultancy firm and we are trying to standardize our project folder structure. We started from the cookiecutter structure which is a great base. However one of the ...
1
vote
1answer
110 views

Where to find height dataset, or datasets in General

Hi there smart people, I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest. For example, I ...
1
vote
2answers
180 views

Null values as useful information for feature engineering

I'm preparing features for a neural network which I'll run in Keras and TensorFlow. The features are generated in Oracle. There I also replace null values. I'm not doing normalization on the database ...
1
vote
0answers
30 views

System administration skills for data engineering [closed]

As a data scientist student, I will do an internship with a lot of data engineering tasks. It would be good if I had some initiation skills in sys-admins. This leads me to my questions: Why should ...
1
vote
3answers
160 views

Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?

It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model ...
0
votes
4answers
80 views

How should we manage large database

We currently manage a large volume of economic data in Excel in my organisation. All of the data is downloaded from different online databases into Excel spreadsheets (one for each data frequency ...
1
vote
1answer
11 views

Text Dataset Help: Need to figure out how to get the publication date of a text

I am a highschool student working on a science fair project in which me and a friend plan to use a neural network for a classification problem. In this case the thing classified will be text and the ...
4
votes
1answer
2k views

What is the difference between ImageNet and ImageNet1k? How to download it?

Some papers mention just ImageNet and some papers mention ImageNet 1k database? What is the difference between these 2? Are they same or is the latter one subset of the former one? I'm working on ...
3
votes
1answer
50 views

How to find the values that never matched in a full join

Hello I have done this full join query: ...
1
vote
1answer
233 views

Machine Learning applied to database design [closed]

Is it possible to apply machine learning to database design given a data source with (pseudo) relationships?
0
votes
1answer
121 views

Will I be able to fetch a Data Analyst job if I learn everything in this image?

India Asking such a question because I am interested learn everything myself and I am confused about what to read. And I am interested by reading books, will that suffice? or should I compulsorily ...
0
votes
1answer
3k views

Replacing column values in pandas with specific column with multiple database operation?

Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can ...
1
vote
0answers
38 views

Database System for Manual Entry

I'm in charge of setting up a patient register (100K+ patients) for a non-profit project with little money. This register should provide the basis for later datascience. I'm not sure how a good ...
1
vote
1answer
422 views

Trying to get Orange connected to Postgres Database

I recently came across Orange, and am very keen to try it for our company's purposes (around 20 potential users, perhaps more). I have installed Orange (latest version) and am trying to connect to our ...
1
vote
0answers
33 views

Database for storing nested s3 objects from broom, sweep, or tidy in R

I'm working on a product that requires automated forecasting of time-series data in R & Shiny. Using R's sweep, broom, & tidy packages, I get these 'tibbles', which are like dataframes with ...
0
votes
2answers
81 views

Database on time to churn in telecomunication

I would like to research about time to churn1 in the telecommunication market. Does anyone have a link to such a database? The only ones I found did not include the time of churn, but only if a ...
1
vote
1answer
778 views

Overcome memory limitation when downloading from database into Orange

I would like to run the association rule mining algorithm of the Orange library on a dataset that is stored in a PostgreSQL database. The table 'buildingset' contains the itemsets for each user, thus ...
2
votes
0answers
2k views

Decode SQLite database blobs: how to start?

Updated: see bottom of this question. Not sure if I am at the right channel here but I have good hopes someone here might be able to help me. I am trying to process and analyse data from a Footscan ...
3
votes
1answer
171 views

Big Data - Data Warehouse Solutions?

I have a dozen of databases that stores different data, and each of them are 100TBs in size. All of the data is stored in AWS services such as RDS, Aurora and Dynamo. Many times I find myself need to ...
1
vote
0answers
3k views

Not enough replicas available for query at consistency all (2 required but only 1 alive)

I have a Cassandra cluster of 2 nodes and a keyspace "test" with a replication factor of 2. The consistency level is All. I am running a spark job which dumps data in Cassandra database. So, the ...
3
votes
1answer
1k views

Is there an overview over recommender system architectures?

I want to learn more about the recommender system topic. I am very interested in the usage of different database systems for this use case. My problem is that I cannot find a good overview over ...
2
votes
1answer
448 views

Are there databases specializing in scientific data

I am currently comparing file formats (HDF5 , etc.) to DBMS systems for a scientific data repository. I know of proprietary solutions such as Oracle extensions, but are there open source/free systems ...
2
votes
0answers
579 views

2D Image-based Avatar Animation With Expressions / Image-To-Image Translation

What I want to do Input an image-sequence (videoframes) that is already cropped and aligned to only the face/hair region and photorealistically replace this Person, Person A (can be anyone, I also ...
0
votes
3answers
3k views

@RISK Vs R. Monte Carlo simulation in R?

Are there any cheaper or open source alternatives to @RISK or are there packages for R that would be able to perform the same tasks? http://www.palisade.com/risk/ @RISK (pronounced “at risk”) ...
2
votes
1answer
36 views

MS Access issue [closed]

I have a database that I send to an excel sheet that auto populates some field in a report with calculations that I need. There are several users and the excel sheet cannot be open at multiple times ...
4
votes
2answers
123 views

Seeking advice on database architecture — given my problem, what tools should I learn?

I'm a fairly experienced R user, but until now I haven't had a good reason to learn to use databases. Now I have a problem where I am dealing with model output that I need to save to disk, and then ...
1
vote
2answers
1k views

Does SQL Server support the Poisson distribution?

I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson ...