Twitter is a great source of information. Using The Health-News-Tweets.zip dataset contains tweets by different agencies like BBC Health, CBC Health, etc. I will perform a clustering on these tweets and will use the elbow method to identify the ideal number of clusters.

My Concern is how i will load only train folder files present in dataset.

Each file contains tweets with ‘|’ as a separator. For ex ID | Timestamp | Tweet My further proceeding will be extracting the tweets column and perform soe preprocesing , tokenize,vectorize,clustering and so but stucked in very beginning .

Any advice Datascience community could give would be much appreciated.


I tried this but getting error

import pyprind
import pandas as pd
import os

progBar = pyprind.ProgBar(3000) # initialize progress bar
df=pd.DataFrame() # empty dataframe
for s in ('test','train'):
    path="./Health-News-Tweets/train/ %s/%s" % (s) #file path
    for file in os.listdir(path):
        with open(os.path.join(path,file),'r') as infile:
            txt = infile.read()
            df= df.append([txt],ignore_index=True) #will append my content of particular text to the dataframe
            progBar.update() # updating progress bar 
            df.columns = ['ID','Timestamp','Tweet']

Error :

TypeError                                 Traceback (most recent call last)
<ipython-input-7-2cabdcb33d40> in <module>
      7 df=pd.DataFrame() # empty dataframe
      8 for s in ('test','train'):
----> 9     path="./Health-News-Tweets/train %s/%s" % (s) #file path
     10     for file in os.listdir(path):
     11         with open(os.path.join(path,file),'r') as infile:

TypeError: not enough arguments for format string

PATH : C:\DATA SCIENCE 2021\LECTURES\FUNDAMENTAL OF LINUX & SERVER CONFIG - MODULE 1\5. MODULE 5 - Machine learning Using Python\Assignment\Health-News-Tweets

  • $\begingroup$ You are trying to use two values in your string as you are using %s twice whereas you are only providing one actual value s. If you want to use the same value twice simply provide it twice using (s, s). $\endgroup$
    – Oxbowerce
    Aug 20 '21 at 10:48
  • $\begingroup$ @Oxbowerce I tried like this path="./Health-News-Tweets/train %s %s" % (s,s) but getting error ` FileNotFoundError: [WinError 3] The system cannot find the path specified: './Health-News-Tweets/train test test' ` . I have provided the data if it could help . $\endgroup$ Aug 20 '21 at 16:58
  • $\begingroup$ That is a completely different error, indicating that the specified filepath does not exist. Make sure that your path is correct and points to an actual location on your computer. $\endgroup$
    – Oxbowerce
    Aug 20 '21 at 17:43
  • $\begingroup$ @Oxbowerce Alright i will try to rectify it . Thank You . $\endgroup$ Aug 20 '21 at 18:06
  • $\begingroup$ You forgot to put a forward slash between the two %s it should be like this path="./Health-News-Tweets/train %s /%s" % (s,s) $\endgroup$
    – user119783
    Aug 21 '21 at 10:24

I tried another method and successfully loaded the required files but unable to do same with above method.

import os
import pandas as pd

#list the files
path = r"C:\DATA SCIENCE 2021\LECTURES\FUNDAMENTAL OF LINUX & SERVER CONFIG - MODULE 1\5. MODULE 5 - Machine learning Using Python\Assignment\Health-News-Tweets\train"
filelist = os.listdir(path) 

#read them into pandas
for file in filelist:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.