First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite.
As for the question itself - you can export your Anaconda environment using:
conda env export > environment.yml
And recreate it using:
conda env create -f environment.yml
Please note that as others suggested - you should use virtual ...
If it's a csv file and you do not need to access all of the data at once when training your algorithm, you can read it in chunks. The pandas.read_csv method allows you to read a file in chunks like this:
import pandas as pd
for chunk in pd.read_csv(<filepath>, chunksize=<your_chunksize_here>)
Here is ...
There are two possibilities: either you need to have all your data in memory for processing (e.g. your machine learning algorithm would want to consume all of it at once), or you can do without it (e.g. your algorithm only needs samples of rows or columns at once).
In the first case, you'll need to solve a memory problem. Increase your memory size, rent a ...
You can manage Spark memory limits programmatically (by the API).
As SparkContext is already available in your Notebook:
You can set as well, but you have to shutdown the existing SparkContext first:
conf = SparkConf().setAppName("App")
conf = (conf.setMaster('local[*]')
See: Graphviz's executables are not found (Python 3.4) and graphviz package doesn't add executable to PATH on windows #1666 and Problem with graphviz #1357 - it's a reoccurring problem (for that program) with the PATH environment variable settings. Installing particular versions, or in a particular order, or manually adding a PATH fixes the problem.
I just had this issue a few days ago! Not sure if this helps in your specific case since you aren't providing so many details, but my situation was to work offline on a 'large' dataset. The data was obtained as 20GB gzipped CSV files from energy meters, time series data at several seconds intervals.
data_root = r"/media/usr/USB STICK"
fname = r"...
First export environment configuration of your current conda environment using:
conda-env export -n your_env_name > your_env_name.yml
conda-env export -n base> base.yml
After running above command their should be yml configuration file in your current directory which contain information of your conda environment
To create new ...
I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it:
That solved my issue. But then I ran into another issue of exceeding max result size of 1024MB. The solution was to add another line in the file above:
There's a much more pythonic solution in pandas...
This takes less than a second on 10 Million rows on my laptop:
for x in X11.E.unique():
Here are the details laid out:
Simple small dataframe -
import numpy as np
import pandas as pd
X11 = pd.DataFrame(np.random.randn(6,4), columns=list('ABCD'))
X11['E'] = [25223, ...
In my case I am able to find graphviz executables manually in anaconda3\Library\bin\graphviz, but I still would get the GraphViz's Executables not found error.
I have unsuccessfully tried zhangqianyuan's suggestion as well as specific orders of module installation and using python-graphviz (official solution, widely discussed here). Only thing I didn't try ...
In my experience, initializing read_csv() with parameter low_memory=False tends to help when reading in large files. I don't think you have mentioned the file type you are reading in, so I am not sure how applicable this is to your situation though.
If your program is mostly Python, you could rely solely on virtual environments.
Create virtual environments to isolate your dependencies rather than using the system libraries. Then use virtual environment tools to duplicate your environments.
In the working virtualenv, create a file with the version of each installed Python library :
pip freeze > ...
Look into 'containers', e.g. Docker (https://www.docker.com/what-container), a more lightweight alternative to virtualization.
It will require some time investment but in the end will provide many benefits.
From the link, where I marked your specific need in bold italic:
Package software into standardized units for development, shipment and deployment
It looks like you want to create dummy variable from a pandas dataframe column. Fortunately, pandas has a special method for it: get_dummies().
Here is a code snippet that you can adapt for your need:
import pandas as pd
data = pd.read_clipboard(sep=',')
#get the names of the first 3 columns
colN = data.columns.values[:3]
#make a copy of the dataframe
You can make sure that command is executed for every terminal (meaning Anaconda will be found) by adding it to your user's bash profile.
Open a terminal and follow these steps:
open the terminal profile: gedit ~/.bashrc
at the end of the file, add: export PATH=~/anaconda3/bin:$PATH
save the file (control+s) and close it
load the changed profile: source ~/....
Try 32-bit Chrome or Firefox, it may significantly lower memory usage.
Remember, it is surprisingly difficult to estimate physical memory consumption of given app: https://dzone.com/articles/windows-process-memory-usage-demystified
Also - unused RAM is wasted RAM. Do not blame browser for taking and using, what's available. When overall memory usage will ...
conda-forge is just an alternative channel where you can upload to or download packages from. Usually it makes no difference where you download the package from but sometimes conda-forge has the latest version.
A wrap up of the existing ways to create an environment based on another one:
Cloning an environment:
From an existing environment:
$ conda create --name ORIG_ENV_NAME --clone CLONE_ENV_NAME
From an exported environment file on the same machine:
$ conda create --name ENV_NAME —-file FILE_NAME.yml
From an exported environment file on a different machine:
What worked for my use case: Generating model diagrams in Django.
But it can also be extended to generate diagrams for any other applications.
I installed the GraphViz for viewing graph from .dot file. Can be installed from graphviz.org.
Create a dot file associated with the project:
python manage.py graph_models -a > dotfile.dot
Or you could create ...
the file .bashrc (hidden file), located in the home directory, runs codes every time a new terminal is opened.
Then add a line on it:
Some other solutions are also provided here.
Activate your conda environment, then use the pip that will also be in your environment (just like the Python interpreter is that of you activated environment).
source activate your_tf_env # now we are in the conda env
which -a pip # should list all pip executables
The first pip one should be in your_tf_env ...
The amazonei environments are for use with the Amazon Elastic Inference accelerators. You can see this mentioned in the DLAMI README file:
for TensorFlow(+Keras2) with Python3 (CUDA 9.0 and Intel MKL-DNN) ______________
____ source activate tensorflow_p36
for Tensorflow(+Amazon Elastic Inference) with Python3 _________________________
____ source activate ...
According to the scikit-learn model persistence docs, it may be better to use joblib instead:
from joblib import dump
from joblib import load
model = load('filename.joblib')