The market 1501 dataset has train, query and gallery folders, each containing multiple views of people from multiple cameras. I would like to understand how to evaluate a model (trained with triplet loss for example), on this dataset. There are multiple things I unfortunately don't understand about this and I have read papers like the "Strong Baseline" paper and tried to look at some code but found it a bit hard to follow.

If I understand correctly, the basic idea is to take a (batch of) query and find the closest examples in the gallery to it. But in Market-1501, the query and gallery have images with the same camera-id and identity. Won't the model just find that image? 2)Does the model have to identify which camera the gallery image was taken from or just the identity?

Do we calculate features for the whole gallery before evaluating or do we have batches of queries and galleries and look for matches in that set? I would really appreciate some of these details as I'm trying to implement a ReID model myself.

I have also posted this on the AI stack exchange as I'm unsure which one is the correct one for this.


1 Answer 1

  1. The images with the same person ID and camera ID are removed for the evaluation calculations
  2. it must only match the identity
  3. I am unsure if there are other implementations but in the one below (from FastReID) it first calculates features for all images, then pairwise distances for all query to all gallery images and then inputs those distances into the evaluation function
def eval_market1501(distmat, q_pids, g_pids, q_camids, g_camids, max_rank):
    """Evaluation with market1501 metric
    Key: for each query identity, its gallery images from the same camera view are discarded.
    num_q, num_g = distmat.shape

    if num_g < max_rank:
        max_rank = num_g
        print('Note: number of gallery samples is quite small, got {}'.format(num_g))

    indices = np.argsort(distmat, axis=1)
    # compute cmc curve for each query
    all_cmc = []
    all_AP = []
    all_INP = []
    num_valid_q = 0.  # number of valid query

    for q_idx in range(num_q):
        # get query pid and camid
        q_pid = q_pids[q_idx]
        q_camid = q_camids[q_idx]

        # remove gallery samples that have the same pid and camid with query
        order = indices[q_idx]
        remove = (g_pids[order] == q_pid) & (g_camids[order] == q_camid)
        keep = np.invert(remove)

        # compute cmc curve
        matches = (g_pids[order] == q_pid).astype(np.int32)
        raw_cmc = matches[keep]  # binary vector, positions with value 1 are correct matches
        if not np.any(raw_cmc):
            # this condition is true when query identity does not appear in gallery

        cmc = raw_cmc.cumsum()

        pos_idx = np.where(raw_cmc == 1)
        max_pos_idx = np.max(pos_idx)
        inp = cmc[max_pos_idx] / (max_pos_idx + 1.0)

        cmc[cmc > 1] = 1

        num_valid_q += 1.

        # compute average precision
        # reference: https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Average_precision
        num_rel = raw_cmc.sum()
        tmp_cmc = raw_cmc.cumsum()
        tmp_cmc = [x / (i + 1.) for i, x in enumerate(tmp_cmc)]
        tmp_cmc = np.asarray(tmp_cmc) * raw_cmc
        AP = tmp_cmc.sum() / num_rel

    assert num_valid_q > 0, 'Error: all query identities do not appear in gallery'

    all_cmc = np.asarray(all_cmc).astype(np.float32)
    all_cmc = all_cmc.sum(0) / num_valid_q

    return all_cmc, all_AP, all_INP

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.