59

First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite. As for the question itself - you can export your Anaconda environment using: conda env export > environment.yml And recreate it using: conda env create -f environment.yml Please note that as others suggested - you should use virtual ...


37

If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows you to read a file in chunks like this: import pandas as pd for chunk in pd.read_csv(<filepath>, chunksize=<your_chunksize_here>) do_processing() train_algorithm() Here is ...


26

There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e.g. your algorithm only needs samples of rows or columns at once). In the first case, you'll need to solve a memory problem. Increase your memory size, rent a ...


17

You can manage Spark memory limits programmatically (by the API). As SparkContext is already available in your Notebook: sc._conf.get('spark.driver.memory') You can set as well, but you have to shutdown the existing SparkContext first: conf = SparkConf().setAppName("App") conf = (conf.setMaster('local[*]') .set('spark.executor.memory', '4G') ...


16

See: Graphviz's executables are not found (Python 3.4) and graphviz package doesn't add executable to PATH on windows #1666 and Problem with graphviz #1357 - it's a reoccurring problem (for that program) with the PATH environment variable settings. Installing particular versions, or in a particular order, or manually adding a PATH fixes the problem. It's ...


10

I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was obtained as 20GB gzipped CSV files from energy meters, time series data at several seconds intervals. File IO: data_root = r"/media/usr/USB STICK" fname = r"...


7

First export environment configuration of your current conda environment using: conda-env export -n your_env_name > your_env_name.yml example: conda-env export -n base> base.yml After running above command their should be yml configuration file in your current directory which contain information of your conda environment To create new ...


6

I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it: spark.driver.memory 14g That solved my issue. But then I ran into another issue of exceeding max result size of 1024MB. The solution was to add another line in the file above: spark.driver.maxResultSize 2g


6

There's a much more pythonic solution in pandas... This takes less than a second on 10 Million rows on my laptop: for x in X11.E.unique(): X11[x]=(X11.E==x).astype(int) X11 Here are the details laid out: Simple small dataframe - import numpy as np import pandas as pd X11 = pd.DataFrame(np.random.randn(6,4), columns=list('ABCD')) X11['E'] = [25223, ...


6

TLDR Use Conda Pack install using conda or pip: conda-forge: conda install -c conda-forge conda-pack PyPI: pip install conda-pack Then for: Backing up: # Pack environment my_env into my_env.tar.gz $ conda pack -n my_env # Pack environment my_env into out_name.tar.gz $ conda pack -n my_env -o out_name.tar.gz # Pack environment located at an ...


5

In my case I am able to find graphviz executables manually in anaconda3\Library\bin\graphviz, but I still would get the GraphViz's Executables not found error. I have unsuccessfully tried zhangqianyuan's suggestion as well as specific orders of module installation and using python-graphviz (official solution, widely discussed here). Only thing I didn't try ...


5

In my experience, initializing read_csv() with parameter low_memory=False tends to help when reading in large files. I don't think you have mentioned the file type you are reading in, so I am not sure how applicable this is to your situation though.


5

If your program is mostly Python, you could rely solely on virtual environments. Create virtual environments to isolate your dependencies rather than using the system libraries. Then use virtual environment tools to duplicate your environments. In the working virtualenv, create a file with the version of each installed Python library : pip freeze > ...


5

Look into 'containers', e.g. Docker (https://www.docker.com/what-container), a more lightweight alternative to virtualization. It will require some time investment but in the end will provide many benefits. From the link, where I marked your specific need in bold italic: Package software into standardized units for development, shipment and deployment A ...


4

It looks like you want to create dummy variable from a pandas dataframe column. Fortunately, pandas has a special method for it: get_dummies(). Here is a code snippet that you can adapt for your need: import pandas as pd data = pd.read_clipboard(sep=',') #get the names of the first 3 columns colN = data.columns.values[:3] #make a copy of the dataframe ...


3

You can make sure that command is executed for every terminal (meaning Anaconda will be found) by adding it to your user's bash profile. Open a terminal and follow these steps: open the terminal profile: gedit ~/.bashrc at the end of the file, add: export PATH=~/anaconda3/bin:$PATH save the file (control+s) and close it load the changed profile: source ~/....


3

Try 32-bit Chrome or Firefox, it may significantly lower memory usage. Remember, it is surprisingly difficult to estimate physical memory consumption of given app: https://dzone.com/articles/windows-process-memory-usage-demystified Also - unused RAM is wasted RAM. Do not blame browser for taking and using, what's available. When overall memory usage will ...


3

conda-forge is just an alternative channel where you can upload to or download packages from. Usually it makes no difference where you download the package from but sometimes conda-forge has the latest version.


2

A wrap up of the existing ways to create an environment based on another one: Cloning an environment: From an existing environment: $ conda create --name ORIG_ENV_NAME --clone CLONE_ENV_NAME From an exported environment file on the same machine: $ conda create --name ENV_NAME —-file FILE_NAME.yml From an exported environment file on a different machine: ...


2

From the very end of this documentation page: Save packages for future use: conda list --export > package-list.txt Reinstall packages from an export file: conda create -n myenv --file package-list.txt


2

Just use the config option when setting SparkSession (as of 2.4) MAX_MEMORY = "5g" spark = SparkSession \ .builder \ .appName("Foo") \ .config("spark.executor.memory", MAX_MEMORY) \ .config("spark.driver.memory", MAX_MEMORY) \ .getOrCreate()


2

If you have Anaconda, you could use Conda manager. Type Conda at Start Panel and try install via Conda. For example: pip3 install graphviz


2

What worked for my use case: Generating model diagrams in Django. But it can also be extended to generate diagrams for any other applications. I installed the GraphViz for viewing graph from .dot file. Can be installed from graphviz.org. Create a dot file associated with the project: python manage.py graph_models -a > dotfile.dot Or you could create ...


2

Write your command in your .bashrc (access at ~/.bashrc ) It will be executed each time you call a terminal.


2

It seems that problem was anyhow with conda installation. I tried pip installation for jupyter and it worked fine. python -m pip install jupyter


2

the file .bashrc (hidden file), located in the home directory, runs codes every time a new terminal is opened. Then add a line on it: export PATH=~/anaconda3/bin:$PATH Some other solutions are also provided here.


2

Activate your conda environment, then use the pip that will also be in your environment (just like the Python interpreter is that of you activated environment). For example: source activate your_tf_env # now we are in the conda env which -a pip # should list all pip executables The first pip one should be in your_tf_env ...


2

The amazonei environments are for use with the Amazon Elastic Inference accelerators. You can see this mentioned in the DLAMI README file: for TensorFlow(+Keras2) with Python3 (CUDA 9.0 and Intel MKL-DNN) ______________ ____ source activate tensorflow_p36 for Tensorflow(+Amazon Elastic Inference) with Python3 _________________________ ____ source activate ...


2

According to the instructions at both Github and Anaconda cloud, you should try conda install -c conda-forge boruta_py


2

According to the scikit-learn model persistence docs, it may be better to use joblib instead: Save model from joblib import dump dump(model, 'filename.joblib') Load model from joblib import load model = load('filename.joblib')


Only top voted, non community-wiki answers of a minimum length are eligible