19
votes
How do I set/get heap size for Spark (via Python notebook)
You can manage Spark memory limits programmatically (by the API).
As SparkContext is already available in your Notebook:
sc._conf.get('spark.driver.memory')
...
- 291
16
votes
Accepted
How to group identical values and count their frequency in Python?
From this sample set I would expect a histogram of receipt that shows
two occurrences of receipt 102857 (since that person bought two items
in one transaction) and one occurrence respectively of ...
- 10.5k
15
votes
How to download a Jupyter Notebook from GitHub?
You can do:
Open the jupyter notebook you want to run.
Click on: Raw
Save Ctrl+S
Remove: .txt
Now navigate to the ...
- 285
12
votes
Issue with IPython/Jupyter on Spark (Unrecognized alias)
IPython has now moved to version 4.0, which means that if you are using it, it will be reading its configuration from ~/.jupyter, not ...
- 281
11
votes
Accepted
How to Use Shap Kernal Explainer with Pipeline models?
The reason is kernel shap sends data as numpy array which has no column names.
so we need to fix it as follows:
...
- 396
11
votes
Generate pdf from jupyter notebook without code
Try this:
jupyter nbconvert --to pdf --TemplateExporter.exclude_input=True my_notebook.ipynb
This also works for html output.
You will find the documentation for ...
- 1,859
7
votes
How do I set/get heap size for Spark (via Python notebook)
I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it:
<...
- 293
6
votes
Accepted
How to download a Jupyter Notebook from GitHub?
If the tutorial is a GitHub repo, sure. Clone the repository. Run jupyter-notebook and open the notebook there.
- 76
5
votes
Accepted
Open-source interactive dashboard in Python
Have a look at IPyWidgets.
I've used it to create interactive dashboards in IPython/Jupyter. It's very concise and powerful.
You define a function:
...
5
votes
Accepted
python print values seasonal_decomposition
seasonal_decompose returns an 'object with seasonal, trend, and resid attributes.' We can access the data by calling the object:
...
- 1,429
5
votes
How to change plot size in nltk.plot()
Try something like this:
import matplotlib.pyplot as plt
plt.figure(figsize=(30, 20)) # the size you want
# your code goes here
- 4,196
4
votes
Accepted
Calculating mean of data frame inside a series object
In the absence of a MultiIndex (the Right Way$^\mathrm{TM}$), the apply method can do what you want; e.g.
...
- 10.5k
4
votes
Accepted
Avoid reloading DataFrame between different python kernels
If it's important for your use cases, you could try switching to Apache Zeppelin.
As all Spark notebooks there share the same Spark context, same Python running environment.
https://zeppelin.apache....
- 198
4
votes
How to export one cell of a jupyter notebook?
You can register a new cell magic, for example:
...
- 41
4
votes
How do I set/get heap size for Spark (via Python notebook)
Just use the config option when setting SparkSession (as of 2.4)
...
- 441
4
votes
Issue with IPython/Jupyter on Spark (Unrecognized alias)
Assume your configure file is ~/.ipython/profile_pyspark/ipython_notebook_config.py, you can still use this configure file by:
...
- 41
4
votes
How to download a Jupyter Notebook from GitHub?
This is a tool that I came across to run Jupyter notebooks: Binder.
You just need to input the repo you are looking at, the branch, and the path. Then you can interactively run the notebook.
The ...
- 3,388
3
votes
How to run a pyspark application in windows 8 command prompt
I'm fairly new to Spark, and have figured out how to integrate with with IPython on Windows 10 and 7. First, check your environment variables for Python and Spark. Here are mine:
SPARK_HOME: C:\spark-...
- 481
3
votes
How to group identical values and count their frequency in Python?
I'm putting together some tutorials around data wrangling. Maybe my jupyter notebook on github will help. I think that it is the key is modifying the line:
...
- 702
3
votes
Accepted
How to plot large web-based heatmaps?
Plotly and Lightning are [supposedly] able to visualize extremely large data sets.
- 146
3
votes
Interactive plots in Jupyter
Plotly is by far the best interactive visualization library/platform I have used, and it works very well with IPython/Jupyter too.
There are tutorials on Plotly's docs which help you integrate it ...
- 8,126
3
votes
Create a binary-classification dataset (python: sklearn.datasets.make_classification)
Let's build some artificial data. There are many ways to do this. I usually always prefer to write my own little script that way I can better tailor the data according to my needs.
Let us first go ...
- 8,688
3
votes
Accepted
Why do a lot of people use ipython notebook over python file when doing analyzing data? Is it the same in industry?
iPython notebooks are great for some cases. I use them because of:
Easy in-place editing and immediate execution, very friendly for quick and experimental stuff
In-place visualization. Also, ability ...
- 806
3
votes
Is my iPython Installation Valid?
It looks fine to me :) the only problem is that your plot (resulting from In [18]) is being displayed on your computer in a separate window somewhere - maybe you ...
- 14.4k
3
votes
Drop all nan value in Series
lat is a series in your data, if you have even only one nan value in your series ,lat, then ...
- 826
3
votes
Accepted
My corr() function in Python keeps resulting in an "ValueError: The truth value of a Series is ambiguous..."
You should look at the documentation. You do not pass the column names as an argument.
subset_df = df[['col1', 'col2']]
subset_df.corr()
That should solve this ...
- 2,330
2
votes
Accepted
How to transpose rows into single column based on time-stamp index in python?
Assuming you have the following source DF:
...
2
votes
Accepted
Jupyter notebook running very slow
In this case you are running a RandomizedSearchCV which is running 100 iterations.
If you consider the fact that for every run of your 30K rows worth of data with youe 300 features (which is a fair ...
- 176
2
votes
How to Use Shap Kernal Explainer with Pipeline models?
I've tried to create a function as suggested but it doesn't work for my code. However, as suggested from an example on Kaggle, I found the below solution:
...
- 21
Only top scored, non community-wiki answers of a minimum length are eligible