2
$\begingroup$

I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields.
I feel that even though there are about 6 million rows (data from 2001 - present) that is a lot of data to drop (especially since the rows contain all other data like crime type, ward, date, location description, etc.)

Here are the columns and the number of NA's found in each column:

COMMUNITY_AREA ID CASE_NUMBER DATE BLOCK IUCR PRIMARY_TYPE DESCRIPTION
        616029  0           0    0     0    0            0           0
LOCATION_DESCRIPTION ARREST DOMESTIC BEAT DISTRICT   WARD FBI_CODE X_COORDINATE
                   0      0        0    0       47 614854        0        60921
Y_COORDINATE YEAR UPDATED_ON LATITUDE LONGITUDE LOCATION CRIME_TYPE COMMUNITY_NAME
       60921    0          0    60921     60921        0          0         616120

When I look up RPubs for this project a lot of people are either dropping all rows with NA's or not even bothering to talk about the missing data, both solutions in my opinion are Not good solutions.

Part of why I don't want to drop all those rows is because each row is a valid crime, when I drop them I am now missing crimes, this contributes to my counts, and categories. And since the data does say what type of crime was committed I can include it in these counts.

Has anyone worked with this data before? Do you have a suggestion for handling the missing data? Or can I leave it in there, is there an issue with that?

I plan on doing a time series analysis on Crime Rate, however, there is no missing data in the crime field so I do not think it will effect it.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Assuming Im reading the format of you table correctly, the NA's are exclusively for geo-location attributes. With "District" being the most complete field with only 47 missing values. Are the locations of certain crimes censored for data privacy reasons?

Of the 6 M crimes in your data set:

  • 10 % or ~600,000 are missing or censored for Ward, Community area or name
  • 1 % or ~60,000 are missing spatial coordinates.

You could determine most of the missing Ward and Community data by analysing the coordinate pairs in a GIS.

How you handle missing data will largely be determined by the hypothesis and analysis you intend to do. You mention Time Series, it should be fine to include all crimes for straight time series at the Chicago level. For higher spatial resolutions, you can do district (dropping 47 Missing points), and so on. How much effort you devote to handling the missing data, such as by using steps outlined above, will depend on how useful that data may be for your analysis. I suspect that most people working with this data tend to drop the NA's as they are mapping the crime rates in different areas.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ There is some amount of privacy: "In order to protect the privacy of crime victims, addresses are shown at the block level only and specific locations are not identified." so the addresses look like: 003XX W LAKE ST $\endgroup$
    – Bear
    Dec 12, 2018 at 20:22
0
$\begingroup$

Echoing BenP, I wonder if you can derive more location info than you think. I see that there are no NAs for the 'beat' variable. Unless I am mistaken, in Chicago a 'beat' is a cop's patrol/responsibility region. So like beat 1 might be the Loop or something like that. You might be able to use that, or other included information, to mitigate your NAs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.