85 votes
Accepted

How to clone Python working environment on another machine?

First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite. As for the question itself - you can export your Anaconda environment using: ...
  • 1,011
40 votes

Opening a 20GB file for analysis with pandas

If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows ...
28 votes

Opening a 20GB file for analysis with pandas

There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e....
  • 2,091
19 votes

How do I set/get heap size for Spark (via Python notebook)

You can manage Spark memory limits programmatically (by the API). As SparkContext is already available in your Notebook: sc._conf.get('spark.driver.memory') ...
  • 291
18 votes

GraphViz not working when imported inside PydotPlus (`GraphViz's executables not found`)

See: Graphviz's executables are not found (Python 3.4) and graphviz package doesn't add executable to PATH on windows #1666 and Problem with graphviz #1357 - it's a reoccurring problem (for that ...
  • 513
18 votes

How to clone Python working environment on another machine?

TLDR Use Conda Pack install using conda or pip: conda-forge: conda install -c conda-forge conda-pack PyPI: ...
  • 425
14 votes

Anconda R version - How to upgrade to 4.0 and later

You need to create a new environment and then you can install R 4.+ in Anaconda. Follow these steps. conda create --name r4-base After activating ...
11 votes

How to clone Python working environment on another machine?

First export environment configuration of your current conda environment using: conda-env export -n your_env_name > your_env_name.yml example: ...
11 votes

Opening a 20GB file for analysis with pandas

I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was ...
9 votes

How to setup and run Conda on Google Colab

Updated Instructions to install Conda on Google Colab Oct 2021 The process is much simpler with condacolab python library Steps Import ...
7 votes

How do I set/get heap size for Spark (via Python notebook)

I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it: <...
  • 293
7 votes

Creating new columns by iterating over rows in pandas dataframe

There's a much more pythonic solution in pandas... This takes less than a second on 10 Million rows on my laptop: ...
  • 6,708
7 votes

How to clone Python working environment on another machine?

If your program is mostly Python, you could rely solely on virtual environments. Create virtual environments to isolate your dependencies rather than using the system libraries. Then use virtual ...
  • 171
6 votes

GraphViz not working when imported inside PydotPlus (`GraphViz's executables not found`)

In my case I am able to find graphviz executables manually in anaconda3\Library\bin\graphviz, but I still would get the ...
  • 61
6 votes

How to clone Python working environment on another machine?

Look into 'containers', e.g. Docker (https://www.docker.com/what-container), a more lightweight alternative to virtualization. It will require some time investment but in the end will provide many ...
  • 1,021
6 votes
Accepted

How to setup and run Conda on Google Colab

Found a way to get Miniconda working in Google colab. For now, use source activate, not conda activate in the 2nd cell. Still working out the bugs with using conda to activate. Full Notebook demo here:...
  • 1,819
5 votes

Opening a 20GB file for analysis with pandas

In my experience, initializing read_csv() with parameter low_memory=False tends to help when reading in large files. I don't ...
  • 166
5 votes

Creating new columns by iterating over rows in pandas dataframe

It looks like you want to create dummy variable from a pandas dataframe column. Fortunately, pandas has a special method for it: get_dummies(). Here is a code ...
  • 306
4 votes

Conda Verification Failed

First try this: conda remove {failing_packages} conda install {failing_packages} Where {failing_packages} is/are the package(...
4 votes
Accepted

Problem using Anaconda

You can make sure that command is executed for every terminal (meaning Anaconda will be found) by adding it to your user's bash profile. Open a terminal and follow these steps: open the terminal ...
  • 14.1k
4 votes

How do I set/get heap size for Spark (via Python notebook)

Just use the config option when setting SparkSession (as of 2.4) ...
  • 441
4 votes

How to clone Python working environment on another machine?

A wrap up of the existing ways to create an environment based on another one: Cloning an environment: From an existing environment: ...
4 votes

What is the lightest browser to launch a jupyter notebook?

Try 32-bit Chrome or Firefox, it may significantly lower memory usage. Remember, it is surprisingly difficult to estimate physical memory consumption of given app: https://dzone.com/articles/windows-...
  • 141
4 votes
Accepted

Difference between AWS amazonei_tensorflow and tensorflow conda environments

The amazonei environments are for use with the Amazon Elastic Inference accelerators. You can see this mentioned in the DLAMI README file: ...
  • 156
3 votes

How to clone Python working environment on another machine?

From the very end of this documentation page: Save packages for future use: conda list --export > package-list.txt Reinstall packages from an export file: <...
3 votes
Accepted

ValueError: pos_label=1 is not a valid label: array(['N', 'Y'], dtype='<U1')

pos_label is an argument of scikit-learn's precision_score (docs); its purpose is, well, to indicate which label is the positive ...
  • 1,820
3 votes
Accepted

normal conda install vs forge install

conda-forge is just an alternative channel where you can upload to or download packages from. Usually it makes no difference where you download the package from but ...
  • 3,760
3 votes
Accepted

No module named "" error when loading the pickle file

According to the scikit-learn model persistence docs, it may be better to use joblib instead: Save model ...
3 votes
Accepted

How to block network access for conda/python environments?

I think you could look at using a Docker image with an Anaconda distribution, which block the network ports unless you specifically open them with the -p option. You could try this page for starters: ...
  • 294
2 votes

How to clone Python working environment on another machine?

One-liner: conda create --clone source_env --name destination_env
  • 141

Only top scored, non community-wiki answers of a minimum length are eligible