Questions tagged [databases]

A comprehensive collection of related data organized for convenient access, generally associated with software to update and query the data.

Filter by
Sorted by
Tagged with
0
votes
1answer
13 views

Do I need to read an entire database for a recommendation system?

Let's say I have a database with approx 100000 rows. I want to build a content-based recommendation system. Do I really need to read the entire database to calculate similarity? That would be very ...
0
votes
0answers
23 views

Decision Tree taking too long to execute

I am training a Decision Tree Regressor on a relatively small data. The dimensions of my train and test sets are (34164, 10) and (8514, 10). Here is the relevant code: ...
2
votes
1answer
304 views

Feature extraction from relational database

In order to build a classifier, I need to extract a few features from the data stored on a MySQL database. I need to join multiple tables and it is taking a lot of ...
0
votes
0answers
17 views

Best image recognition API to implement for eCommerce Lifestyle/Sculpture site

I'm planning an eCommerce site currently. We are likely running WooCommerce and looking to implement Algolia for our search features. We feel that for our particular purposes, a visual search would be ...
1
vote
2answers
214 views

ML model deployment architecture?

I came from a software development background and we have separate servers of the same database (dev, test, prod). The reason for this is because we develop our apps against the dev DB, run tests ...
10
votes
5answers
3k views

What is the ideal database that allows fast cosine distance?

I'm currently trying to store many feature vectors in a database so that, upon request, I can compare an incoming feature vector against many other (if not all) stored in the db. I would need to ...
1
vote
1answer
25 views

How to work with hundreds of CSVs with millions of rows in each?

So I'm doing a project on the COVID-19 Tweets dataset from the IEEE port and I plan to analyse the tweets over the time period from March 2020 till date. The thing is there's more than 300 CSVs for ...
0
votes
0answers
9 views

Data Warehouse design schema for sales (items, shopping carts, ..)

I have to design some OLAP-Cubes based on an OLTP database for the sales. At first, that sounds very basic and common but I struggle with the item <-> shopping cart thing. One cart can contain ...
0
votes
1answer
54 views

Azure Cloud SQL - Querying large number of rows with Python

I have a Python Flask application that connects to an Azure Cloud SQL Database, and uses the Pandas read_sql method with SQLAlchemy to perform a select operation on a table and load it into a ...
0
votes
1answer
15 views

query category wise dara

I have stored the some products and there respective category in sql. Like this. Now I want fetch all product from table with respective category. output should be=>
0
votes
0answers
8 views

Database best-practice when sunsetting a feature

Say, your product team decided to sunset a feature of your product, and there are columns in your database related to that feature. Is it a good practice to keep the columns (so you don't lose this ...
1
vote
2answers
29 views

How can I create a table from an existing table in SQL but using cells from the old table as columns in the new table? [closed]

I have a table, and I want to create a new table such as the one below (from the table above) In SQL, I tried using the following commands. I am able to generate a table with only one column like ...
0
votes
0answers
22 views

What should be the architecture design for the dataset to be used for a machine learning API

I have aroud 30GB of market dataset in csv format which is downloaded onto my server computer everyday. Now I want to use this dataset to do some computations and provide some analysis. The user will ...
2
votes
1answer
20 views

DBMS or Software for privacy sensitive data

We have a dataset of very privacy sensitive people data and want to build a database with it. The data protection department in our company doesn't like the idea that the data scientists are able to ...
1
vote
0answers
9 views

Is high school GPA included in IPEDs NCES databases?

I’m currently developing a computer tool which will sort through the US college and university NCES data provided in the IPEDs website to match users with potential universities. NCES shows university ...
0
votes
0answers
25 views

Working with mySQL and pytorch Dataset

I am working with high frequency time indexed data. We have 2 types of data each with about 5 columns. For each type of data we have 2500 streams coming in and being updated every 1ms-100ms (the ...
2
votes
0answers
11 views

Is there a name for the discipline of creating database structures (tables, columns, etc) to model a domain?

I develop software, mostly web applications. I generally use a backend framework like Ruby on Rails. I have been relatively successful at modeling business domain structures into code and into ...
1
vote
0answers
78 views

Database augmentation using Sobel filter (aka gradient)

I'm working in PyTorch and to improve a ProGAN training I'm augmenting a database by calculating, for each image, the horizontal and vertical gradient via the Sobel operator. Do you have any reading ...
3
votes
1answer
1k views

Is there an overview over recommender system architectures?

I want to learn more about the recommender system topic. I am very interested in the usage of different database systems for this use case. My problem is that I cannot find a good overview of ...
0
votes
0answers
30 views

Why does a simple query succeed in mysql but fail in druid?

I have a mysql database table with only one column called __time which stores a time stamp. The following query works perfectly and will return the ...
1
vote
1answer
25 views

I Have Issues Installing Basemap

I tried to install Basemap and it gives me this: preparing transaction: done verifying transaction: done executing transaction: failed ERROR conda.core.link:_execute(507): An error occurred while ...
34
votes
5answers
21k views

Best practices to store Python machine learning models

What are the best practices to save, store, and share machine learning models? In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ...
2
votes
2answers
155 views

Data repositories like UCI

Are there any other data repositories like UCI and mlData, for biological data?? I want to know about mostly biological data set.
4
votes
2answers
823 views

What is the most used format to save data with type information

I am exporting data from an SQL database and importing it into R. This is a two step process since I first (automatically) download the data to a hard drive and then import the file with R. ...
0
votes
2answers
91 views

Filenotfounderror: [errno 2] no such file or directory: 'nsr001.ecg'

I am trying to save contents of physiobank Normal Sinus Rhythm RR Interval Database into a numpy array but I keep getting an error: ...
0
votes
0answers
21 views

Image Dataset for course project

I am currently studying Machine Learning course and now we are requested ( each student ) to find an Image dataset to work on for the course project. I am new to ML and it's now a week and I am trying ...
1
vote
1answer
655 views

Connect Orange 3.20 to postgresql database

I installed orange 3.20 on windows 7. It works so far, the problem is connecting it to a server based Postgres database. While the connection can be made in the moment, when you try to load a table ...
0
votes
0answers
9 views

How to store and query biometric data for an authentication system?

I am trying to design, and hopefully implement, an authentication system which centers around the use of biometric images. I plan to use different machine learning and deep learning techniques to help ...
3
votes
3answers
1k views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
2
votes
2answers
3k views

Data set with multiple tables [closed]

I want to check an algorithm that learns the connections between different tables in a database. Where can I find datasets that are compound of multiple tables (i.e. more than 5) that have ...
6
votes
2answers
160 views

ETL and Data Engineering - is it purely the knowledge of tools or is there theory behind it?

I would like to better understand what a good Data Englineer must know or what he does. Job descriptions mostly list tools that are required, such as Python. If it is possible to separate Data ...
9
votes
3answers
5k views

Human activity recognition using smartphone data set problem

I'm new to this community and hopefully my question will well fit in here. As part of my undergraduate data analytics course I have choose to do the project on human activity recognition using ...
10
votes
3answers
7k views

Which is faster: PostgreSQL vs MongoDB on large JSON datasets?

I have a large dataset with 9m JSON objects at ~300 bytes each. They are posts from a link aggregator: basically links (a URL, title and author id) and comments (text and author ID) + metadata. They ...
0
votes
0answers
15 views

Machine Learning Recommender high data intensity

We are building a recommender engine to be integrated in an app that, each time an API is called, will pull thousands of records from an Azure SQL database and create recommendations. Currently with ...
13
votes
5answers
15k views

Advantages of pandas dataframe to regular relational database

In Data Science, many seem to be using pandas dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to regular relational databases like MySQL, which ...
50
votes
9answers
24k views

How to deal with version control of large amounts of (binary) data

I am a PhD student of Geophysics and work with large amounts of image data (hundreds of GB, tens of thousands of files). I know svn and ...
1
vote
1answer
2k views

How to plot time series data from SQLite database

I am trying to plot time series data by accessing the SQLite database. I successfully connect the database by coding the below script,however, i couldn't figure out how to plot the time series data ...
2
votes
1answer
111 views

What is the best practice for data folder structuring?

I work for a small data science consultancy firm and we are trying to standardize our project folder structure. We started from the cookiecutter structure which is a great base. However one of the ...
1
vote
1answer
371 views

Where to find height dataset, or datasets in General

Hi there smart people, I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest. For example, I ...
1
vote
2answers
360 views

Null values as useful information for feature engineering

I'm preparing features for a neural network which I'll run in Keras and TensorFlow. The features are generated in Oracle. There I also replace null values. I'm not doing normalization on the database ...
1
vote
0answers
31 views

System administration skills for data engineering [closed]

As a data scientist student, I will do an internship with a lot of data engineering tasks. It would be good if I had some initiation skills in sys-admins. This leads me to my questions: Why should ...
1
vote
3answers
200 views

Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?

It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model ...
0
votes
4answers
93 views

How should we manage large database

We currently manage a large volume of economic data in Excel in my organisation. All of the data is downloaded from different online databases into Excel spreadsheets (one for each data frequency ...
1
vote
1answer
11 views

Text Dataset Help: Need to figure out how to get the publication date of a text

I am a highschool student working on a science fair project in which me and a friend plan to use a neural network for a classification problem. In this case the thing classified will be text and the ...
6
votes
1answer
5k views

What is the difference between ImageNet and ImageNet1k? How to download it?

Some papers mention just ImageNet and some papers mention ImageNet 1k database? What is the difference between these 2? Are they same or is the latter one subset of the former one? I'm working on ...
3
votes
1answer
50 views

How to find the values that never matched in a full join

Hello I have done this full join query: ...
1
vote
1answer
274 views

Machine Learning applied to database design [closed]

Is it possible to apply machine learning to database design given a data source with (pseudo) relationships?
0
votes
1answer
126 views

Will I be able to fetch a Data Analyst job if I learn everything in this image?

India Asking such a question because I am interested learn everything myself and I am confused about what to read. And I am interested by reading books, will that suffice? or should I compulsorily ...
0
votes
1answer
3k views

Replacing column values in pandas with specific column with multiple database operation?

Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can ...
1
vote
0answers
49 views

Database System for Manual Entry

I'm in charge of setting up a patient register (100K+ patients) for a non-profit project with little money. This register should provide the basis for later datascience. I'm not sure how a good ...