18
You can manage Spark memory limits programmatically (by the API).
As SparkContext is already available in your Notebook:
sc._conf.get('spark.driver.memory')
You can set as well, but you have to shutdown the existing SparkContext first:
conf = SparkConf().setAppName("App")
conf = (conf.setMaster('local[*]')
.set('spark.executor.memory', '4G')
...
16
From this sample set I would expect a histogram of receipt that shows
two occurrences of receipt 102857 (since that person bought two items
in one transaction) and one occurrence respectively of receipt 102856
and of receipt 102858.
Then you want:
df.groupby('receipt').receipt.count()
receipt
102856 1
102857 2
102858 1
Name: receipt, ...
12
IPython has now moved to version 4.0, which means that if you are using it, it will be reading its configuration from ~/.jupyter, not ~/.ipython. You have to create a new configuration file with
jupyter notebook --generate-config
and then edit the resulting ~/.jupyter/jupyter_notebook_config.py file according to your needs.
More installation instructions ...
11
You can do:
Open the jupyter notebook you want to run.
Click on: Raw
Save Ctrl+S
Remove: .txt
Now navigate to the directory where notebook is downloaded in jupyter notebook and open it.
10
The reason is kernel shap sends data as numpy array which has no column names.
so we need to fix it as follows:
def model_predict(data_asarray):
data_asframe = pd.DataFrame(data_asarray, columns=feature_names)
return estimator.predict(data_asframe)
Then,
shap_kernel_explainer = shap.KernelExplainer(model_predict, x_train, link='logit')
...
6
I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it:
spark.driver.memory 14g
That solved my issue. But then I ran into another issue of exceeding max result size of 1024MB. The solution was to add another line in the file above:
spark.driver.maxResultSize 2g
6
If the tutorial is a GitHub repo, sure. Clone the repository. Run jupyter-notebook and open the notebook there.
6
Try this:
jupyter nbconvert --to pdf --TemplateExporter.exclude_input=True my_notebook.ipynb
This also works for html output.
You will find the documentation for this and other options here.
FYI, for complex notebooks, this may generate errors depending on your version of nbconvert, LaTeX and other components. In that case try to convert to html then print ...
5
Have a look at IPyWidgets.
I've used it to create interactive dashboards in IPython/Jupyter. It's very concise and powerful.
You define a function:
def f(x):
return x
And you connect it to a widget using the Interact() function.
interact(f, x=10);
This generates a widget (a slider in this case) automatically and updates your function with new input when ...
5
seasonal_decompose returns an 'object with seasonal, trend, and resid attributes.' We can access the data by calling the object:
res = seasonal_decompose(series, model='additive', freq=365)
residual = res.resid
seasonal = res.seasonal
trend = res.trend
print trend
etc...
4
In the absence of a MultiIndex (the Right Way$^\mathrm{TM}$), the apply method can do what you want; e.g.
df.assign(
max_retweet=df.tweets.apply(lambda x: x.retweet_count.argmax('retweet_count')),
avg_retweet=df.tweets.apply(lambda x: x.retweet_count.mean())
)
avg_retweet max_retweet
26662 0.045476 187
32316 0.821538 ...
4
Try something like this:
import matplotlib.pyplot as plt
plt.figure(figsize=(30, 20)) # the size you want
# your code goes here
4
If it's important for your use cases, you could try switching to Apache Zeppelin.
As all Spark notebooks there share the same Spark context, same Python running environment.
https://zeppelin.apache.org/
So what you're asking happens natively in Zeppelin. Or to be complete, it is an option to share the same Spark context / same Python envrionment between ...
4
Assume your configure file is ~/.ipython/profile_pyspark/ipython_notebook_config.py, you can still use this configure file by:
ipython notebook --config='~/.ipython/profile_pyspark/ipython_notebook_config.py'
or
jupyter-notebook --config='~/.ipython/profile_pyspark/ipython_notebook_config.py'
3
I'm fairly new to Spark, and have figured out how to integrate with with IPython on Windows 10 and 7. First, check your environment variables for Python and Spark. Here are mine:
SPARK_HOME: C:\spark-1.6.0-bin-hadoop2.6\
I use Enthought Canopy, so Python is already integrated in my system path. Next, launch Python or IPython and use the following code. If ...
3
I have found some solution and will post it here, because somebody, who works with graphlab, can have the same question.
We can look at the example here: Six degrees of Kevin Bacon
At te beginning of the program execution you need to run next command:
graphlab.canvas.set_target('ipynb')
Exactly this is a key of the whole problem (at least by me:-)
At ...
3
Just use the config option when setting SparkSession (as of 2.4)
MAX_MEMORY = "5g"
spark = SparkSession \
.builder \
.appName("Foo") \
.config("spark.executor.memory", MAX_MEMORY) \
.config("spark.driver.memory", MAX_MEMORY) \
.getOrCreate()
3
I'm putting together some tutorials around data wrangling. Maybe my jupyter notebook on github will help. I think that it is the key is modifying the line:
df.groupby('male')['age'].mean()
to be:
df.groupby('reciept')['prod_name'].count()
To group by multiple variables this should work:
df.groupby(['reciept','date'])['reciept'].count()
3
Plotly and Lightning are [supposedly] able to visualize extremely large data sets.
3
Plotly is by far the best interactive visualization library/platform I have used, and it works very well with IPython/Jupyter too.
There are tutorials on Plotly's docs which help you integrate it with Jupyter. Another tutorial.
3
You can register a new cell magic, for example:
from IPython.core.magic import register_cell_magic
@register_cell_magic
def run_and_save(line, cell):
'Run and save python code block to a file'
with open(line, 'wt') as fd:
fd.write(cell)
code = compile(cell, line, 'exec')
exec(code, globals())
Now, you can use the run_and_save magic:
%...
3
This is a tool that I came across to run Jupyter notebooks: Binder.
You just need to input the repo you are looking at, the branch, and the path. Then you can interactively run the notebook.
The notebook gets hosted by the website, so you don't need to worry about computing power on your machine or installing all the required packages.
3
Let's build some artificial data. There are many ways to do this. I usually always prefer to write my own little script that way I can better tailor the data according to my needs.
Let us first go through some basics about data. A lot of the time in nature you will find Gaussian distributions especially when discussing characteristics such as height, skin ...
3
iPython notebooks are great for some cases. I use them because of:
Easy in-place editing and immediate execution, very friendly for quick and experimental stuff
In-place visualization. Also, ability to have multiple figures on the same page, compare them, re-run figures, move the cells. Much more convenient than multiple and independent OpenCV's imshow ...
3
It looks fine to me :) the only problem is that your plot (resulting from In [18]) is being displayed on your computer in a separate window somewhere - maybe you have to find it. Once you close that window, your iPython prompt woill return to In [19]. You could alternatively press Ctrl-C in the iPython session, but this will end the session.
If the problem ...
3
lat is a series in your data, if you have even only one nan value in your series ,lat, then you will loose all of your series.
Another problem might be in the lat column values might not be suitable for converting to pd.to_numeric such as "1,2" (not "1.2"), "a", "nan" etc. Then you will loose your series again.
Therefore:
Check:
len(lat.dropna()) == len(...
2
Assuming you have the following source DF:
In [21]: df
Out[21]:
Time val1 val2 val3
0 2017-11-17 11 12 13
1 2017-11-18 24 25 0
2 2017-11-19 37 0 0
Solution:
In [22]: (df.replace(0, np.nan)
.set_index('Time', append=True)
.stack()
.reset_index()
.rename(columns={0:'...
2
In this case you are running a RandomizedSearchCV which is running 100 iterations.
If you consider the fact that for every run of your 30K rows worth of data with youe 300 features (which is a fair amount), you would be looking at an average run time of ~ 1.2 minutes per run.
You could however speed this if you were running thia via GPUs instead of CPUs as ...
2
I've tried to create a function as suggested but it doesn't work for my code. However, as suggested from an example on Kaggle, I found the below solution:
import shap
#load JS vis in the notebook
shap.initjs()
#set the tree explainer as the model of the pipeline
explainer = shap.TreeExplainer(pipeline['classifier'])
#apply the preprocessing to x_test
...
Only top voted, non community-wiki answers of a minimum length are eligible