Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [hierarchical-data-format]

The tag has no usage guidance.

1
vote
0answers
24 views

Advice on dealing with very large datasets - HDF5, Python

I've recently started working on an application for visualization of really big datasets. While reading online it became apparent that most people use HDF5 for storing big, multi-dimensional datasets ...
1
vote
0answers
53 views

Best classification technique for following kind of data set

I have a large table where each record or row represents a single salesperson, and there are 50 columns or dimensions where each column represents one of 50 products potentially sold by any given ...
-1
votes
1answer
14 views

Finding the best “depth” of ICD9 codes with pseudo-hierarchical clustering

Here is a common problem in health care modeling. Did I just invent a new algorithm or has someone already thought of this? The goal is to find the most homogeneous partition of patients by medical ...
1
vote
1answer
26 views

What is the best to identify the proper hierarchy of this data?

So I worked on a hierarchical clustering algorithm to be able to determine which items are most similar, and what attributes are most important. I have two tables: Table 1: contains a bunch of item ...
1
vote
0answers
32 views

Using an ontology to recognize named entities in free text

I'm trying to solve a fairly basic problem in NPL efficiently. What tool or software package would you use to identify the words, or group of words that are part of an given ontology within a free ...
1
vote
1answer
193 views

Using PCA to cluster multidimensional data (RFM variables)

So i am performing k-means clustering on RFM variables (Recency, Frequency, Monetary). The RFM variables are in the form of quantiles (1-4). I used PCA and found the PCA components. I then used the ...
0
votes
0answers
72 views

Clustering of devices in locations?

My question is about using some sort of AI to assess if devices are located in any of a list of venues. I'd ask of machine learning, but so far we're doing this with an expert system, and we are ...
2
votes
0answers
104 views

Using neural networks for classification in Hierarchical data

I am stuck in a problem wherein I have hierarchical data, say $A->B->C$(smaller to biggest), and the smallest unit of data is a block(A consists of multiple blocks, B consists of multiple A's, ...
0
votes
1answer
37 views

How to treat input that inherently has a tree structure?

When having a single vector you use an MLP neural network When having a 2D structure you use a CNN neural network When having a sequence you use a RNN neural network Now you have preprocessed an ...
1
vote
1answer
1k views

Machine learning for predicting HTML Elements on a web page?

My goal is to implement an assistant for crawling web data for users that don't understand anything about HTML or DOM. I will show a web page to the user and the user has to select, what data he is ...
3
votes
1answer
535 views

Derivation of Ward's formula for agglomerative clustering

Figure describe formula of ward method for hierarchical or agglomarative clustering i.e. increment of total error after merging two cluster. How did they get $$\frac{n_A\cdot n_B}{n_A+n_B} ||\vec ...
1
vote
0answers
50 views

Forecasting daily usage of a prouct

Lets say that you have panel data, of daily consumption of product p1 of 10000 individuals. The panel data is on a daily basis for only one month, this means that $t=1,...,30$. The question is how do ...
2
votes
0answers
55 views

How to mitigate the hierarchical error propagation in tree-structured classification

Suppose we have a multi-class classification problem, where the number of classes $K \geq 3$ We use a tree structure of multiple SVMs to divide and conquer the problem, with one example in the figure ...
1
vote
0answers
433 views

What are the strategies for feature engineering in a hierarchical/relational structured data?

I have a feature set with each feature having its own feature-set, and they are laid out in a SQL database with foreign key relationships. The target is defined on the top level feature-set and is a ...
2
votes
0answers
57 views

Spatial clustering of data points on a grid to obtain variable resolution map with constant statistical confidence

I have a grey valued image which is calculated as the mean of a series of images. The value of each pixel is therefore associated to a standard error. The pixel values and the relative standard error ...
1
vote
2answers
212 views

Complete link clustering

I'm conjecturing that with Complete-linkage clustering two elements from the same cluster will always be closer to each other some other element from another cluster. In more formal terms: Let $C$ ...
6
votes
2answers
1k views

Efficient dynamic clustering

I have a set of datapoints from the unit interval (i.e. 1-dimensional dataset with numerical values). I receive some additional datapoints online, and moreover the value of some datapoints might ...
26
votes
3answers
7k views

Hierarchical Data Format. What are the advantages compared to alternative formats?

What are the main benefits from storing data in HDF? And what are the main data science tasks where HDF is really suitable and useful?