Clustering objects defined by vector

I have a set of objects (98 total). I need to cluster these objects based on their pairwise distance. Each pair contains approximately 2000 values (not consistent).

This is what I have done so far. I took the average for each pair and then built a distance matrix among all objects. Then applied hierarchical clustering. However, although the range of values defining the objects have very wide range (0.1 - 0.8), but taking the average making all those score ~0.5 which producing terrible clusters.

So, what could be a better measurement of distribution than just taking arithmetic average? Or, what clustering algorithm is useful if I want to define each object considering all the values rather than taking only one value (i.e. average) to define the whole object?