76

First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite. As for the question itself - you can export your Anaconda environment using: conda env export > environment.yml And recreate it using: conda env create -f environment.yml Please note that as others suggested - you should use virtual ...


40

If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows you to read a file in chunks like this: import pandas as pd for chunk in pd.read_csv(<filepath>, chunksize=<your_chunksize_here>) do_processing() train_algorithm() Here is ...


28

There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e.g. your algorithm only needs samples of rows or columns at once). In the first case, you'll need to solve a memory problem. Increase your memory size, rent a ...


18

See: Graphviz's executables are not found (Python 3.4) and graphviz package doesn't add executable to PATH on windows #1666 and Problem with graphviz #1357 - it's a reoccurring problem (for that program) with the PATH environment variable settings. Installing particular versions, or in a particular order, or manually adding a PATH fixes the problem. It's ...


18

You can manage Spark memory limits programmatically (by the API). As SparkContext is already available in your Notebook: sc._conf.get('spark.driver.memory') You can set as well, but you have to shutdown the existing SparkContext first: conf = SparkConf().setAppName("App") conf = (conf.setMaster('local[*]') .set('spark.executor.memory', '4G') ...


13

TLDR Use Conda Pack install using conda or pip: conda-forge: conda install -c conda-forge conda-pack PyPI: pip install conda-pack Then for: Backing up: # Pack environment my_env into my_env.tar.gz $ conda pack -n my_env # Pack environment my_env into out_name.tar.gz $ conda pack -n my_env -o out_name.tar.gz # Pack environment located at an ...


11

I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was obtained as 20GB gzipped CSV files from energy meters, time series data at several seconds intervals. File IO: data_root = r"/media/usr/USB STICK" fname = r"...


10

First export environment configuration of your current conda environment using: conda-env export -n your_env_name > your_env_name.yml example: conda-env export -n base> base.yml After running above command their should be yml configuration file in your current directory which contain information of your conda environment To create new ...


10

You need to create a new environment and then you can install R 4.+ in Anaconda. Follow these steps. conda create --name r4-base After activating r4-base run these commands conda activate r4-base conda install -c conda-forge r-base conda install -c conda-forge/label/gcc7 r-base Finally, you will notice r-basa version 4 will be installed. Thereafter, you ...


7

There's a much more pythonic solution in pandas... This takes less than a second on 10 Million rows on my laptop: for x in X11.E.unique(): X11[x]=(X11.E==x).astype(int) X11 Here are the details laid out: Simple small dataframe - import numpy as np import pandas as pd X11 = pd.DataFrame(np.random.randn(6,4), columns=list('ABCD')) X11['E'] = [25223, ...


7

If your program is mostly Python, you could rely solely on virtual environments. Create virtual environments to isolate your dependencies rather than using the system libraries. Then use virtual environment tools to duplicate your environments. In the working virtualenv, create a file with the version of each installed Python library : pip freeze > ...


6

In my case I am able to find graphviz executables manually in anaconda3\Library\bin\graphviz, but I still would get the GraphViz's Executables not found error. I have unsuccessfully tried zhangqianyuan's suggestion as well as specific orders of module installation and using python-graphviz (official solution, widely discussed here). Only thing I didn't try ...


6

Look into 'containers', e.g. Docker (https://www.docker.com/what-container), a more lightweight alternative to virtualization. It will require some time investment but in the end will provide many benefits. From the link, where I marked your specific need in bold italic: Package software into standardized units for development, shipment and deployment A ...


6

I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it: spark.driver.memory 14g That solved my issue. But then I ran into another issue of exceeding max result size of 1024MB. The solution was to add another line in the file above: spark.driver.maxResultSize 2g


5

It looks like you want to create dummy variable from a pandas dataframe column. Fortunately, pandas has a special method for it: get_dummies(). Here is a code snippet that you can adapt for your need: import pandas as pd data = pd.read_clipboard(sep=',') #get the names of the first 3 columns colN = data.columns.values[:3] #make a copy of the dataframe ...


5

In my experience, initializing read_csv() with parameter low_memory=False tends to help when reading in large files. I don't think you have mentioned the file type you are reading in, so I am not sure how applicable this is to your situation though.


4

A wrap up of the existing ways to create an environment based on another one: Cloning an environment: From an existing environment: $ conda create --name NEW_ENV_NAME --clone ORIG_ENV_NAME From an exported environment file on the same machine: $ conda create --name ENV_NAME —-file FILE_NAME.yml From an exported environment file on a different machine: $...


4

You can make sure that command is executed for every terminal (meaning Anaconda will be found) by adding it to your user's bash profile. Open a terminal and follow these steps: open the terminal profile: gedit ~/.bashrc at the end of the file, add: export PATH=~/anaconda3/bin:$PATH save the file (control+s) and close it load the changed profile: source ~/....


4

Try 32-bit Chrome or Firefox, it may significantly lower memory usage. Remember, it is surprisingly difficult to estimate physical memory consumption of given app: https://dzone.com/articles/windows-process-memory-usage-demystified Also - unused RAM is wasted RAM. Do not blame browser for taking and using, what's available. When overall memory usage will ...


4

The amazonei environments are for use with the Amazon Elastic Inference accelerators. You can see this mentioned in the DLAMI README file: for TensorFlow(+Keras2) with Python3 (CUDA 9.0 and Intel MKL-DNN) ______________ ____ source activate tensorflow_p36 for Tensorflow(+Amazon Elastic Inference) with Python3 _________________________ ____ source activate ...


4

Found a way to get Miniconda working in Google colab. For now, use source activate, not conda activate in the 2nd cell. Still working out the bugs with using conda to activate. Full Notebook demo here: https://donaldsrepo.github.io/Notebooks/GoogleColabCondaCreateEnv.html github with demo notebook: https://github.com/donaldsrepo/SampleNotebooks/blob/master/...


3

First try this: conda remove {failing_packages} conda install {failing_packages} Where {failing_packages} is/are the package(s) for which an error is reported. If that fails, you could try forcing an over-install (this solution is very handy and will likely fix many issues): conda install -f {failing_packages} So, for example, for the original poster of ...


3

Just use the config option when setting SparkSession (as of 2.4) MAX_MEMORY = "5g" spark = SparkSession \ .builder \ .appName("Foo") \ .config("spark.executor.memory", MAX_MEMORY) \ .config("spark.driver.memory", MAX_MEMORY) \ .getOrCreate()


3

From the very end of this documentation page: Save packages for future use: conda list --export > package-list.txt Reinstall packages from an export file: conda create -n myenv --file package-list.txt


3

conda-forge is just an alternative channel where you can upload to or download packages from. Usually it makes no difference where you download the package from but sometimes conda-forge has the latest version.


3

According to the scikit-learn model persistence docs, it may be better to use joblib instead: Save model from joblib import dump dump(model, 'filename.joblib') Load model from joblib import load model = load('filename.joblib')


3

I think you could look at using a Docker image with an Anaconda distribution, which block the network ports unless you specifically open them with the -p option. You could try this page for starters: https://hub.docker.com/u/continuumio


2

One-liner: conda create --clone source_env --name destination_env


2

the file .bashrc (hidden file), located in the home directory, runs codes every time a new terminal is opened. Then add a line on it: export PATH=~/anaconda3/bin:$PATH Some other solutions are also provided here.


2

Write your command in your .bashrc (access at ~/.bashrc ) It will be executed each time you call a terminal.


Only top voted, non community-wiki answers of a minimum length are eligible