Skip to main content

Questions tagged [data-formats]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

What to include in fact and dimension table from election database

I am working with election dataset of India of year 2014 and data for 2019 I also have table for party names and descriptions and finally state name with code. I am not getting how do i create a ...
Sugam Sharma's user avatar
0 votes
1 answer
186 views

How to drop the previous rows of a database based on a matching value in a column?

So I am currently trying to sort through a data frame containing attribute classes and values of teams. However, my data has multiple rows of different classes and values of the same Team ID/Attribute ...
DaddyMuffin's user avatar
1 vote
1 answer
3k views

Date time conversion in a CSV column [closed]

I am new to data science. I am attempting to write a program using regression techniques, and all of my values are numerical, except for the date and time (UTC), which are written in this format: HH:...
RocketBlaster05's user avatar
1 vote
1 answer
23 views

Is there any way to analyze the format of text strings? [closed]

I have a lot of data which basically consists of alphanumeric text on individual lines which can very in length and contain delimiters. Since there are many thousands of lines of text, I'm looking to ...
cosmarchy's user avatar
  • 111
3 votes
2 answers
898 views

Storing Large dataset for processing and analysis of data

I am new to data engineering and wanted to know , what is the best way to store more than 3000 GB of data for further processing and analysis ? I am specifically looking for open source resources . I ...
user14519285's user avatar
0 votes
1 answer
168 views

Python: convert variables into correct format for DataFrame

I have 3 variables that I would like to use to build my dataset but since they are in a weird shape/format, I had no success so far. I'm quite new to this and really appreciate any help!! The 3 ...
newbieeeee_e's user avatar
2 votes
1 answer
3k views

How to store efficiently very large sparse 3D matrices

To train a CNN, I have stacked arrays of images over observations [observations x width x length]. The dataset is very sparse ($95\%$). What would be an efficient ...
hH1sG0n3's user avatar
  • 2,058
0 votes
1 answer
488 views

Running a query in R after establishing dbconnect

I do not seem to figure out what is wrong it the following statement. The connection to the DWH is established but the query statement in R seems not to work, with the following error : ...
ColRow's user avatar
  • 53
1 vote
2 answers
54 views

Converting data format

I'm trying to use the recent COVID-19 data from the site of Italian Civil Protection, but they use a rather complicated time format that I'm finding troublesome as a novice to plot as data in a graph. ...
Leonardo Becarelli's user avatar
1 vote
1 answer
103 views

Best file format for transfer of EHR data

I am working on a clinical trial where we have several sites sending us EHR data. The sites are currently sending the data in excel files. I have a feeling someone's opening the files because 3 of ...
Sam Weisenthal's user avatar
1 vote
1 answer
39 views

Containing multicomponent data in rows or columns

I have been working with DNA sequences and compiled a table with features from those sequences. I have a column called Trimer, which contains strings. For some DNA sequences there is one trimer of ...
hko's user avatar
  • 11
3 votes
1 answer
8k views

Getting stock data in a discipline manner from Yahoo finance

I used the below code for downloading stock data from yahoo finance:- ...
coding_ninza's user avatar
0 votes
1 answer
710 views

NCHW vs NHWC in Machine Learning

As I've been introducing myself to the various deep learning frameworks, I've noticed a difference in the default placement of channels for images. Is there a substantial difference between NCHW vs ...
CovertKoala's user avatar
1 vote
1 answer
3k views

.h5 file format does not close properly

import h5py #added hf = h5py.File('../images.h5', 'w') #added hf.close() #added h5_file = tables.open_file("images.h5", mode="w") I also tried: ...
Fatemeh Asgarinejad's user avatar
1 vote
0 answers
34 views

Labeling data as having an error?

I am curating a large quantity of data from different sensors. If I know that a particular sensor was broken or poorly calibrated for a particular time range, what would be a useful way of annotating ...
David LeBauer's user avatar
8 votes
2 answers
128k views

ValueError: could not convert string to float: '���'

I have a (2M, 23) dimensional numpy array X. It has a dtype of <U26, i.e. unicode string ...
cappy0704's user avatar
  • 231
1 vote
0 answers
52 views

Need help with confusing dataset formats for Images and annotations

I have a more general question, which I could not answer after quite a bit of google. Actually, I got more confused. I hope you can share some light. I am familiar with deep learning, Keras but come ...
Florida Man's user avatar
1 vote
2 answers
84 views

Python list formatting

I have a list, which looks like this: params ['h\x00i\x00', '\x00t\x00h\x00e\x00r\x00e\x00'] Now, all I want is to merge these two elements into the string "hi ...
user96931's user avatar
  • 113
1 vote
1 answer
37 views

Prepare data : many products per row

I would like to find how to change my data structure to make it compatible with a ML model, with the following constraints : A Product is defined by {id, price, continuousVariable1, ..., ...
pushStack's user avatar
  • 113
0 votes
1 answer
992 views

converting column from a factor to date format YYYY-MM-DDThh:mm:ss.SSSZ

i want to convert the whole column from a factor to date. The str of the dataset I am using is given below: ...
maira khan's user avatar
1 vote
1 answer
349 views

parquet format: advise on log content

I'm using a python script to log IO of a grid job. the log is formatted like this: ...
Kiwy's user avatar
  • 125
0 votes
1 answer
502 views

Formatting Data for use in SKmultilearn

I have a dataframe of X values and one with the y values in what I think is the proper format. I took the single feature y and converted it to dummies so it's binary and is m rows by n features. <...
jma's user avatar
  • 143
1 vote
1 answer
56 views

Format for time series data with non-trivial sampled data

I have a data stream that I would like to share with some data scientists. It is a regularly captured time series with some fields that are simple scalars, booleans. Each sample has a UTC time and ...
JeffV's user avatar
  • 113
3 votes
3 answers
3k views

Building a Citation Network to Analyze in R

I am working on network analysis project at one of our Nation's Service Academies, and I need a little help. As a starting point, we are looking at a citation network that we build by using the ...
Joseph Bosse's user avatar
1 vote
1 answer
1k views

Format for X_train in keras using theano

I want to try out Keras (Theano backend) for regressions after already using sklearn. For this I uses this nice tutorial http://machinelearningmastery.com/regression-tutorial-keras-deep-learning-...
El Burro's user avatar
  • 800
0 votes
2 answers
1k views

Connecting Infusionsoft data to Google data studio

I want to create a Google Data studio dashboard from Infusion soft data. The main problem are the connectors - there are multiple tools that provide direct connectors but they are paid solutions like ...
Pravin's user avatar
  • 111
1 vote
1 answer
401 views

R data frame create a new variable which corresponds to one of the existing one

Currently this is what I get. "community_area_clean" is the new variable that I added by extracting a variable that lists all the communities from a data frame. "CommArea_Name" is the original "...
user avatar
4 votes
2 answers
1k views

What is the most used format to save data with type information

I am exporting data from an SQL database and importing it into R. This is a two step process since I first (automatically) download the data to a hard drive and then import the file with R. ...
Pieter's user avatar
  • 961
1 vote
2 answers
169 views

Are ICD codes forward compatible?

The ICD code scheme has gone through many revisions: https://en.wikipedia.org/wiki/International_Statistical_Classification_of_Diseases_and_Related_Health_Problems Is there any compatibility between ...
user48956's user avatar
  • 121
2 votes
3 answers
4k views

What is the best file format to store an uncompressed 2D matrix?

For what it's worth my particular case is a symmetrical matrix, but this question should be answered more generally.
Austin Capobianco's user avatar
1 vote
1 answer
43 views

Given the files and their parsed content how can I reverse engineer the file format?

I have access to a few thousand files binary files and I don't have the program that generated them. The files have been parsed and I also have access to the database where the parsed fields are ...
ramana_k's user avatar
  • 131
2 votes
1 answer
94 views

Advantage of a treebank in XML format

Which treebanks are based on an XML format? What is the advantage of XML format for a treebank? I think it may have effects on annotation and querying the treebank. for example LASSY and Alpino or ...
Ahmad's user avatar
  • 407
2 votes
2 answers
527 views

Why ConLL is not in XML format

I am new to NLP, I found a format named ConLL which seems a tab-separated file, like ID FORM LEMMA PLEMMA POS PPOS FEAT PFEAT HEAD PHEAD DEPREL PDEPREL I found ...
Ahmad's user avatar
  • 407
1 vote
3 answers
2k views

Transposing Every nth row to column in a large dataset

I am attempting to work with a very large data-set (~1.5mil lines) for the first time in SAS and I am having some difficulty. The data-set I have is formatted as a "long" .txt file as follows: ...
roboallen's user avatar
1 vote
2 answers
854 views

Appropriate way to store data in R

I have data, which looks like this: These data are only for one subject. I will have a lot more. These data will be analyzed in R. Now I'm storing them like this: ...
AndriusZ's user avatar
  • 145
0 votes
1 answer
212 views

Best format for recording time stamp and GPS

In most data acquisition settings it is useful to tag your data with time and location. If I write the data to csv file, what are the best formats that I can use for this two variables if I want to ...
ontan's user avatar
  • 13
40 votes
4 answers
27k views

What are the advantages of HDF compared to alternative formats?

What are the advantages of HDF compared to alternative formats? What are the main data science tasks where HDF is really suitable and useful?
IgorS's user avatar
  • 5,474