Hot answers tagged

4

There are 2 things you can do here: 1.) Use libraries like Dask to speed up your data preprocessing. Here is the link 2.) Use cloud computing services like Azure, AWS or GCP. I am not aware of other two but I have worked on Azure and it provides a lot of options for implementing a data science solution. You get options like Auto-ML, Azure Designer, Python ML ...


3

You can register a new cell magic, for example: from IPython.core.magic import register_cell_magic @register_cell_magic def run_and_save(line, cell): 'Run and save python code block to a file' with open(line, 'wt') as fd: fd.write(cell) code = compile(cell, line, 'exec') exec(code, globals()) Now, you can use the run_and_save magic: %...


3

A million observations of 20 features should be very manageable on a laptop, if a little slow. Cloud computing for very large datasets is staggeringly expensive and offers little or no benefit unless and until you have good parallelization in place. I would recommend keeping that option as your last resort. For the initial data exploration and ...


3

Assuming you cannot add more memory to your computer (or free up some of the memory), you could try 2 general approaches: Read only some of the data into memory e.g. a subset of the rows or columns. reduce the precision of the data from float64 to float32. From your error, it looks like you are loading data into a numpy array, so somewhere in your code, ...


2

Updated Instructions to install Conda on Google Colab Oct 2021 The process is much simpler with condacolab python library Steps Import condacolab python library Install condacolab !pip install -q condacolab import condacolab condacolab.install() Post kernel restart, check condacolab installation import condacolab condacolab.check() Environment You can ...


2

I do not see your whole example but this usually happens when you have not initialized your classifier. Even more, to test, you first have to train your classifier (e.g. clf().fit(X_train, y_train)).


2

I am using both almond and toree. For executing only scala code, I am using almond. If you want to learn new features in scala (ex: 2.13), then use almond(Add Scala 2.13.4 support in almond v0.11.0). Almond might start supporting scala 3.0 once it is released. For spark, I am using toree. Toree works fine for me. I am using toree 0.5.0 with spark 3.0.0. ...


1

Parallelize your analyses on a single (multi-cpu) machine with e.g. pandarallel or the like or go for broke with scala/spark/hadoop if the problem wont fit on a single machine.


1

I think commitstrip got it spot on. :-) For myself, I would invest into soft skills, i.e. working with people, decoding what is needed from what is requested, planning large projects, and into hard skills like statistics, information theory and related areas of maths. Coding is just means to get things done IMHO


1

Make sure you are working with Qt Console (anaconda): Install Jupiter extensions: !pip install jupyter_contrib_nbextensions !jupyter contrib nbextension install --user Enable nbextension: !jupyter nbextension enable codefolding/main Install pyppeteer: !python -m pip install -U notebook-as-pdf !pyppeteer-install MAKE SURE YOUR WORKING DIRECTORY IS WHERE ...


1

For Question 1, replace val_acc with val_accuracy since the metric is named as accuracy. This might also solve your 2nd question. ... filepath="weights-improvement-{epoch:02d}-{val_accuracy:.2f}.hdf5" checkpoint = ModelCheckpoint(filepath, monitor= "val_accuracy" , verbose=1, save_best_only=True, mode= "max" ) ...


1

I have had the same problem while training in huge data sets in Jupyter Notebooks. The only solution I found was to create a scrip .py with my training process (including model persistence) and running it from my terminal (python3 myscript.py)


1

You can change the Chrome shortcut like this: In System Preferences > Keyboard > Shortcuts > App Shortcuts, click the [+] button, and select Google Chrome as the Application. Put Bookmarks->Bookmark All Tabs... in the Menu Title and a different Keyboard Shortcut.


1

Product Manager from Deepnote here. I'm sorry it doesn't perform well when the notebook is too big. We're aware of the issue, and it's quite hard to solve, but I'll stress its importance to our engineers. I'm not sure how Datalore handles large notebooks. When you compare them, they both have some nice features (collaboration, secure integrations, reactivity)...


1

You are only seeing the last one since you overwrite your object df at each iteration and when the loop is finished it will store only the last value store which happens to be mumbai If you want to create a DataFrame with all the results you should do something like this: ldf = list() india = ["chennai","pune","mumbai"] for city ...


1

You may use df.groupby(['BirthDate', 'ZipCode']).size().reset_index().rename(columns={0: 'n'}) and now you have a data frame that you can easily manipulate.


1

Your data is multilabel: each row can have more than one label. But sklearn's Naive Bayes doesn't support that format of problem. You can use MultiOutputClassifier to wrap the Naive Bayes classifier, effectively training one model for each of the labels. See the User Guide, especially the MultiOutput classification section. You may also want to consider the ...


1

As of October 2020... In terms of basic neural network functionality, they are pretty equivalent. Some differences: Stability: tensorflow 2.0 underwent a lot of changes from tensorflow 1.x, specifically in the very way it worked (they changed from a computational graph paradigm to an imperative paradigm). This caused a lot of friction and left many ...


1

Here is the latest work on JupyterLab. Shared editing with collaborative notebooks in JupyterLab To try it → install the alpha release: 3.1.0a7 & set the flag --collaborative https://github.com/jupyterlab/jupyterlab/pull/10118


1

I am sure you have found your answer by now, but for others. Setup from mpl_toolkits.mplot3d import axes3d import matplotlib.pyplot as plt import numpy as np import pandas as pd from matplotlib import style data_dic = {2001 : [15, 23, 24], 2002 : [16, 25, 23], 2003 : [14, 18, 22], 2004 : [18, 24, 26]} df = pd.DataFrame(...


1

You can heat "Kernel" and Choose "Restart & Run All". Then you do not need to run your codes line by line!


Only top voted, non community-wiki answers of a minimum length are eligible