Of course, making them smaller will cause loss of information. Since before you had N x M data points for each image to describe the content of the image, but after resizing you will have n x m (Where n and N denote the number of rows in an image, m and M denotes the number of columns in an image. Also, n <= N and/or m <= M). That is, a small number of pixels will result in less information.
Open CV's resize() function squeezes the image instead of cropping, meaning it preserves aspect ratio, original content but it displays objects or parts in the image with fewer pixels. To achieve it, Open CV uses the interpolation technique. (read more here)
Two solutions you can try:
If your smallest image's size does not differ much from the size of the majority of your image, then resize them to the smallest. Because in that case, if you have a good model, your performance will not be affected severely.
If your smallest image's size differs much from the majority, or in general you have a group of small images and big images which sizes differ a lot, then choose a threshold of minimum size to which you will resize the images and drop the images that have a size less than your threshold.
In both cases, you will lose data or information, it is inevitable. The best practice is to test various possibilities to see which works well. Look at the distribution of the sizes of your images, make your decision accordingly.
If the size of the smallest image is big enough, and the smallest image and the biggest image do not differ a lot in size, I would just resize them. Because it would at least, gives a chance to the model to capture the feature in the image. However, in the second case (dropping the small images), you would not even give that chance to the model.