Questions tagged [databases]

A comprehensive collection of related data organized for convenient access, generally associated with software to update and query the data.

Filter by
Sorted by
Tagged with
1
vote
0answers
23 views

where to store embeddings for similarity search?

I've asked on stackoverflow already (here), but I figured that the approach of storing embeddings in an ordinary postgres-Database might be flawed from the very beginning. I will shortly etch out the ...
1
vote
1answer
50 views

How to plot time series data from SQLite database

I am trying to plot time series data by accessing the SQLite database. I successfully connect the database by coding the below script,however, i couldn't figure out how to plot the time series data ...
0
votes
0answers
11 views

How to save a trained ml model(e.g. decision tree) in database and load it using Flask to predict user submitted data?

I have tried the following way. It worked, but it is giving a different prediction in Flask vs Jupyter (which is correct) ...
0
votes
0answers
10 views

NL2SQL task, if we have enough data, what will the model achieve for hard SQL?

We are afraid that the hard SQL like TABLE JOIN is the limit for industrial application. Addition info: https://yale-lily.github.io/spider Thank you very much.
0
votes
0answers
7 views

DBMS - Indexed Sequential Files

If i already have a primary index on a file, what is the purpose of ordering also the records of the file by primary key ? Can't i simply use the primary index to get a fast access to records without ...
0
votes
0answers
5 views

DBMS - Heap file organization for Secondary Indices

In my book about Database Management Systems the authors say that the Heap file organization (the records are put in the disk blocks following the inserting order) is usually associated with secondary ...
2
votes
1answer
26 views

What is the best practice for data folder structuring?

I work for a small data science consultancy firm and we are trying to standardize our project folder structure. We started from the cookiecutter structure which is a great base. However one of the ...
0
votes
0answers
7 views

Utility of using a Primary Index in a relational dbms

If the records stored in each block are ordered by the primary key, what is the vantage of using a primary index? In fact, without a primary index, I still can do a binary search: taking the block in ...
0
votes
0answers
13 views

Minimal Fragmentation in Distributed Database

In a distributed database, using selection we get a minimal fragmentation if If all the predicates of a set Pr are relevant, Pr is minimal. Given fragments generated by a set $M(P)$, we say a ...
1
vote
1answer
23 views

Where to find height dataset, or datasets in General

Hi there smart people, I am new to data Science and wanted to take my first few steps. Unfortunately I struggle to find datasets or any data at all regarding my topics of interest. For example, I ...
0
votes
0answers
6 views

Is there an ISO or other standard or best practice for structuring records related to organisations?

I'm working on a project which includes redesigning the way an application represents data about organisations. The current design is quite "flat" - meaning for example that if I want to add a new ...
0
votes
0answers
8 views

What would be the best data store for fast write/read of billions of float32s? SQLite? Hdf5? Other?

I am trying to create a library for sparse training. So it would need fast read/write of not only the machine learning model weights, but their optimizer momentum values as well. At the moment I would ...
1
vote
2answers
36 views

Null values as useful information for feature engineering

I'm preparing features for a neural network which I'll run in Keras and TensorFlow. The features are generated in Oracle. There I also replace null values. I'm not doing normalization on the database ...
0
votes
0answers
14 views

Any there databases with native support for applying a NN model to produce search rankings?

The Situation: I have a simple neural net with an input vector that consists of the euclidean distance between the attributes of two cars. (For example the attributes would include wheel size, car ...
0
votes
0answers
19 views

How to Model Real-World Historical Data Containing Temporal Periods

I have reached one of those points in research where I do not know enough about what I do not know to find possible solutions. I am working with a dataset containing information on federal judicial ...
0
votes
0answers
22 views

Best way to store and combine live data in visualization project

I am new to setting up databases beyond simple SQL and MongoDb, but have technical proficiency in programming and networking, primarily in Javascript. I am tasked with collecting data at multiple ...
0
votes
0answers
24 views

How best to find patterns in data that exhibit one-to-many and many-to-one relations?

I have a lot of log data related to alerts on a computer platform. Each log entry represents an alert, but several of the same alert can be fired for a single event over a span of time (either because ...
1
vote
0answers
25 views

System administration skills for data engineering [closed]

As a data scientist student, I will do an internship with a lot of data engineering tasks. It would be good if I had some initiation skills in sys-admins. This leads me to my questions: Why should ...
1
vote
2answers
83 views

ML model deployment architecture?

I came from a software development background and we have separate servers of the same database (dev, test, prod). The reason for this is because we develop our apps against the dev DB, run tests ...
1
vote
1answer
10 views

Text Dataset Help: Need to figure out how to get the publication date of a text

I am a highschool student working on a science fair project in which me and a friend plan to use a neural network for a classification problem. In this case the thing classified will be text and the ...
1
vote
3answers
83 views

Why are pandas/R/etc so common in a data scientist or analyst's workflow when one can resort to UDFs in an RDBMS?

It seems common for an analyst to have this workflow when using an RDBMS: use SQL to get a subset of data from the database, export it or use a connector, and then apply a data mining / model ...
0
votes
4answers
74 views

How should we manage large database

We currently manage a large volume of economic data in Excel in my organisation. All of the data is downloaded from different online databases into Excel spreadsheets (one for each data frequency ...
1
vote
1answer
158 views

Connect Orange 3.20 to postgresql database

I installed orange 3.20 on windows 7. It works so far, the problem is connecting it to a server based Postgres database. While the connection can be made in the moment, when you try to load a table ...
3
votes
1answer
317 views

What is the difference between ImageNet and ImageNet1k? How to download it?

Some papers mention just ImageNet and some papers mention ImageNet 1k database? What is the difference between these 2? Are they same or is the latter one subset of the former one? I'm working on ...
3
votes
1answer
44 views

How to find the values that never matched in a full join

Hello I have done this full join query: ...
4
votes
3answers
585 views

What is the ideal database that allows fast cosine distance?

I'm currently trying to store many feature vectors in a database so that, upon request, I can compare an incoming feature vector against many other (if not all) stored in the db. I would need to ...
1
vote
0answers
10 views

To what extent can one link data in a structured or rigorous way, given only partial knowledge base?

I seem to have stumbled on an interesting problem: suppose I am given partial information in two tables, say imports and exports between different countries, and I can query these, but e.g. the ...
1
vote
1answer
114 views

Machine Learning applied to database design [closed]

Is it possible to apply machine learning to database design given a data source with (pseudo) relationships?
0
votes
1answer
98 views

Will I be able to fetch a Data Analyst job if I learn everything in this image?

India Asking such a question because I am interested learn everything myself and I am confused about what to read. And I am interested by reading books, will that suffice? or should I compulsorily ...
0
votes
1answer
3k views

Replacing column values in pandas with specific column with multiple database operation?

Need to replace database of the column in specific refine query with multiple operations as mention in below image. Trying such operation as an individual, but can't understand which method to use can ...
1
vote
0answers
20 views

Database System for Manual Entry

I'm in charge of setting up a patient register (100K+ patients) for a non-profit project with little money. This register should provide the basis for later datascience. I'm not sure how a good ...
2
votes
0answers
31 views

Database for storing nested s3 objects from broom, sweep, or tidy in R

I'm working on a product that requires automated forecasting of time-series data in R & Shiny. Using R's sweep, broom, & tidy packages, I get these 'tibbles', which are like dataframes with ...
0
votes
2answers
64 views

Database on time to churn in telecomunication

I would like to research about time to churn1 in the telecommunication market. Does anyone have a link to such a database? The only ones I found did not include the time of churn, but only if a ...
1
vote
0answers
24 views

Local U.S. City Government Expenditures APIs?

I'm currently working on a project that will compile the various expenditures of local city governments (Health, Transportation, etc), what initiatives are present in those major cities, and more ...
0
votes
1answer
267 views

Trying to get Orange connected to Postgres Database

I recently came across Orange, and am very keen to try it for our company's purposes (around 20 potential users, perhaps more). I have installed Orange (latest version) and am trying to connect to our ...
3
votes
1answer
119 views

Big Data - Data Warehouse Solutions?

I have a dozen of databases that stores different data, and each of them are 100TBs in size. All of the data is stored in AWS services such as RDS, Aurora and Dynamo. Many times I find myself need to ...
1
vote
0answers
2k views

Not enough replicas available for query at consistency all (2 required but only 1 alive)

I have a Cassandra cluster of 2 nodes and a keyspace "test" with a replication factor of 2. The consistency level is All. I am running a spark job which dumps data in Cassandra database. So, the ...
2
votes
1answer
1k views

Is there an overview over recommender system architectures?

I want to learn more about the recommender system topic. I am very interested in the usage of different database systems for this use case. My problem is that I cannot find a good overview over ...
1
vote
1answer
422 views

Are there databases specializing in scientific data

I am currently comparing file formats (HDF5 , etc.) to DBMS systems for a scientific data repository. I know of proprietary solutions such as Oracle extensions, but are there open source/free systems ...
2
votes
0answers
537 views

2D Image-based Avatar Animation With Expressions / Image-To-Image Translation

What I want to do Input an image-sequence (videoframes) that is already cropped and aligned to only the face/hair region and photorealistically replace this Person, Person A (can be anyone, I also ...
2
votes
1answer
34 views

MS Access issue [closed]

I have a database that I send to an excel sheet that auto populates some field in a report with calculations that I need. There are several users and the excel sheet cannot be open at multiple times ...
11
votes
4answers
9k views

Advantages of pandas dataframe to regular relational database

In Data Science, many seem to be using pandas dataframes as the datastore. What are the features of pandas that make it a superior datastore compared to regular relational databases like MySQL, which ...
24
votes
3answers
9k views

Best practices to store Python machine learning models

What are the best practices to save, store, and share machine learning models? In Python, we generally store the binary representation of the model, using pickle or joblib. Models, in my case, can be ...
4
votes
2answers
117 views

Seeking advice on database architecture — given my problem, what tools should I learn?

I'm a fairly experienced R user, but until now I haven't had a good reason to learn to use databases. Now I have a problem where I am dealing with model output that I need to save to disk, and then ...
-1
votes
1answer
25 views

How to document operations on a database? [closed]

I am working on a scientific article, where I am going to work with a large SQL based database. This work consist of everything from washing data, creating sub-databases etc. How can I document this ...
1
vote
2answers
960 views

Does SQL Server support the Poisson distribution?

I am trying to do some work that is primarily based in SQL Server. I cannot seem to find a native function to SQL that supports the Poisson distribution. Has anyone had success with applying Poisson ...
1
vote
1answer
96 views

Is there a term for “this month last year” in a report? [closed]

I'm building a report that has month over month data, but also "this month last year". Is there a better/standard way of describing this?
1
vote
3answers
1k views

Instead of one-hot encoding, can I store the same information in one column using a single value?

Rather than creating 15 additional columns full of sparse binary data, could I: 1) use the first 15 prime numbers as indexes for the 15 categories 2) store data by multiplying the prime numbers of ...
2
votes
2answers
6k views

Classified Enron Email Dataset

I'm building a system able to classify emails into different categories (positive, negative, out of office, etc...) and I'm looking for a dataset of already classified emails to avoid hand ...
0
votes
1answer
169 views

is there big difference between data Science , big Data and database? [closed]

is there big difference between data Science , big Data and database? i am confused in these three can anyone help me to out of this confusion?