Hot answers tagged

19 votes

How do I set/get heap size for Spark (via Python notebook)

You can manage Spark memory limits programmatically (by the API). As SparkContext is already available in your Notebook: sc._conf.get('spark.driver.memory') ...
  • 291
16 votes
Accepted

How to group identical values and count their frequency in Python?

From this sample set I would expect a histogram of receipt that shows two occurrences of receipt 102857 (since that person bought two items in one transaction) and one occurrence respectively of ...
  • 10.5k
15 votes

How to download a Jupyter Notebook from GitHub?

You can do: Open the jupyter notebook you want to run. Click on: Raw Save Ctrl+S Remove: .txt Now navigate to the ...
  • 285
12 votes

Issue with IPython/Jupyter on Spark (Unrecognized alias)

IPython has now moved to version 4.0, which means that if you are using it, it will be reading its configuration from ~/.jupyter, not ...
  • 281
11 votes
Accepted

How to Use Shap Kernal Explainer with Pipeline models?

The reason is kernel shap sends data as numpy array which has no column names. so we need to fix it as follows: ...
11 votes

Generate pdf from jupyter notebook without code

Try this: jupyter nbconvert --to pdf --TemplateExporter.exclude_input=True my_notebook.ipynb This also works for html output. You will find the documentation for ...
  • 1,859
7 votes

How do I set/get heap size for Spark (via Python notebook)

I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it: <...
  • 293
6 votes
Accepted

How to download a Jupyter Notebook from GitHub?

If the tutorial is a GitHub repo, sure. Clone the repository. Run jupyter-notebook and open the notebook there.
5 votes
Accepted

Open-source interactive dashboard in Python

Have a look at IPyWidgets. I've used it to create interactive dashboards in IPython/Jupyter. It's very concise and powerful. You define a function: ...
5 votes
Accepted

python print values seasonal_decomposition

seasonal_decompose returns an 'object with seasonal, trend, and resid attributes.' We can access the data by calling the object: ...
  • 1,429
5 votes

How to change plot size in nltk.plot()

Try something like this: import matplotlib.pyplot as plt plt.figure(figsize=(30, 20)) # the size you want # your code goes here
  • 4,196
4 votes
Accepted

Calculating mean of data frame inside a series object

In the absence of a MultiIndex (the Right Way$^\mathrm{TM}$), the apply method can do what you want; e.g. ...
  • 10.5k
4 votes
Accepted

Avoid reloading DataFrame between different python kernels

If it's important for your use cases, you could try switching to Apache Zeppelin. As all Spark notebooks there share the same Spark context, same Python running environment. https://zeppelin.apache....
  • 198
4 votes

How to export one cell of a jupyter notebook?

You can register a new cell magic, for example: ...
4 votes

How do I set/get heap size for Spark (via Python notebook)

Just use the config option when setting SparkSession (as of 2.4) ...
  • 441
4 votes

Issue with IPython/Jupyter on Spark (Unrecognized alias)

Assume your configure file is ~/.ipython/profile_pyspark/ipython_notebook_config.py, you can still use this configure file by: ...
  • 41
4 votes

How to download a Jupyter Notebook from GitHub?

This is a tool that I came across to run Jupyter notebooks: Binder. You just need to input the repo you are looking at, the branch, and the path. Then you can interactively run the notebook. The ...
3 votes

How to run a pyspark application in windows 8 command prompt

I'm fairly new to Spark, and have figured out how to integrate with with IPython on Windows 10 and 7. First, check your environment variables for Python and Spark. Here are mine: SPARK_HOME: C:\spark-...
  • 481
3 votes

How to group identical values and count their frequency in Python?

I'm putting together some tutorials around data wrangling. Maybe my jupyter notebook on github will help. I think that it is the key is modifying the line: ...
  • 702
3 votes
Accepted

How to plot large web-based heatmaps?

Plotly and Lightning are [supposedly] able to visualize extremely large data sets.
  • 146
3 votes

Interactive plots in Jupyter

Plotly is by far the best interactive visualization library/platform I have used, and it works very well with IPython/Jupyter too. There are tutorials on Plotly's docs which help you integrate it ...
  • 8,126
3 votes

Create a binary-classification dataset (python: sklearn.datasets.make_classification)

Let's build some artificial data. There are many ways to do this. I usually always prefer to write my own little script that way I can better tailor the data according to my needs. Let us first go ...
  • 8,688
3 votes
Accepted

Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?

iPython notebooks are great for some cases. I use them because of: Easy in-place editing and immediate execution, very friendly for quick and experimental stuff In-place visualization. Also, ability ...
3 votes

Is my iPython Installation Valid?

It looks fine to me :) the only problem is that your plot (resulting from In [18]) is being displayed on your computer in a separate window somewhere - maybe you ...
  • 14.4k
3 votes

Drop all nan value in Series

lat is a series in your data, if you have even only one nan value in your series ,lat, then ...
3 votes
Accepted

My corr() function in Python keeps resulting in an "ValueError: The truth value of a Series is ambiguous..."

You should look at the documentation. You do not pass the column names as an argument. subset_df = df[['col1', 'col2']] subset_df.corr() That should solve this ...
  • 2,330
2 votes
Accepted

How to transpose rows into single column based on time-stamp index in python?

Assuming you have the following source DF: ...
2 votes
Accepted

Jupyter notebook running very slow

In this case you are running a RandomizedSearchCV which is running 100 iterations. If you consider the fact that for every run of your 30K rows worth of data with youe 300 features (which is a fair ...
  • 176
2 votes

How to Use Shap Kernal Explainer with Pipeline models?

I've tried to create a function as suggested but it doesn't work for my code. However, as suggested from an example on Kaggle, I found the below solution: ...
  • 21

Only top scored, non community-wiki answers of a minimum length are eligible