77
$\begingroup$

I developed a machine learning model with Python (Anaconda + Flask) on my workstation and all goes well. Later, I tried to ship this program onto another machine where of course I tried to set up the same environment, but the program fails to run. I copied the program to other machines where it also runs smoothly.

I cannot figure out what the problem is in the failed case (both the program code and the error message are copious so I am not able to present them here) but I'm almost certain that it is something with the different versions of the dependencies.

So, my question is that given an environment where a certain program runs well, how can I clone it to another where it should run well also? Of course, without the cloning of the full system ;)

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Use conda env export. $\endgroup$
    – kbrose
    Commented Oct 26, 2017 at 14:04

10 Answers 10

Reset to default
92
$\begingroup$

First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite.

As for the question itself - you can export your Anaconda environment using:

conda env export > environment.yml

And recreate it using:

conda env create -f environment.yml

Please note that as others suggested - you should use virtual environments which allows you to create a certain environment that is separated from that of your machine and manage it more easily.

To create a virtual environment in Anaconda you can use:

conda create -n yourenvname python=x.x anaconda

which you activate using:

source activate yourenvname
Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ The Anaconda documentation is not very clear on wether to use conda create or conda env create when sharing/recreating an environment. Could you please further detail why you recommend using conda env create in this situation? $\endgroup$
    – Tanguy
    Commented Nov 15, 2018 at 10:05
  • $\begingroup$ You can find some notes about the difference between conda create and conda env create here: groups.google.com/a/continuum.io/forum/#!topic/conda/… That said, I think you could usually use them interchangeably. $\endgroup$
    – ginge
    Commented Nov 15, 2018 at 13:21
  • 1
    $\begingroup$ I have seen this thread, but precisely I am trying to understand precisely in which situation each option (conda create vs conda env create) should be prefered and what are the downsides of each (e.g. : "[conda env create is for] environments in which packages using pip have been installed into, which causes additional complexity" : what kind of additional complexity does it add?). $\endgroup$
    – Tanguy
    Commented Nov 15, 2018 at 13:52
  • $\begingroup$ I wanna ask that when running conda env create -f environment.yml, this will cause error because the name of virtenv in yml file has already been used. Change the name to your new virtenv to overcome. $\endgroup$
    – Giang Nguyen
    Commented Apr 21, 2019 at 9:10
  • $\begingroup$ This way saves your life! if you create an environment from a file based on Anaconda Managing Environments page instructions, doesn't work if you use another platform. conda list --explicit > FILE_NAME exports binaries for the current platform and apparently not working on another one. $\endgroup$
    – Shawn Azdam
    Commented Jul 12, 2019 at 20:33
24
$\begingroup$

TLDR
Use Conda Pack
install using conda or pip:

conda-forge: 

conda install -c conda-forge conda-pack

PyPI: 

pip install conda-pack

Then for:

Backing up: 

# Pack environment my_env into my_env.tar.gz
$ conda pack -n my_env

# Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz

# Pack environment located at an explicit path into my_env.tar.gz
$ conda pack -p /explicit/path/to/my_env

Restoring:

And to restore it on the other machine(s):

# Unpack environment into directory `my_env`
$ mkdir -p my_env
$ tar -xzf my_env.tar.gz -C my_env

# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
$ ./my_env/bin/python

# Activate the environment. This adds `my_env/bin` to your path
$ source my_env/bin/activate

# Run Python from in the environment
(my_env) $ python

# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack

a bit of an explanation :

If you plan on getting an exact copy of your current environment and then move it to another machine with the same platform and OS, without redownloading all packages again from Internet (good for offline machines/behind firewalls). All other previous methods require internet connection. so in case you dont have access to internet, then you can use conda pack.

Conda Pack

Conda-pack is a command line tool that archives a conda environment, which includes all the binaries of the packages installed in the environment. This is useful when you want to reproduce an environment with limited or no internet access. All the previous methods download packages from their respective repositories to create an environment. Keep in mind that conda-pack is both platform and operating system specific and that the target computer must have the same platform and OS as the source computer.

To install conda-pack, make sure you are in the root or base environment so that it is available in sub-environments. Conda-pack is available at conda-forge or PyPI.

for future updates check the ref

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ This solution is ideal for cloning an environment to a machine with no connection to the internet. $\endgroup$
    – albarji
    Commented Mar 3, 2020 at 12:08
  • 1
    $\begingroup$ For me the conda-unpack did not add the my_env to the conda's list of virtual env. I had to simply copy the my_env under the Anaconda3\env path, then it worked. $\endgroup$
    – TwinPenguins
    Commented Jun 30, 2020 at 15:48
  • $\begingroup$ This solution is also excellent for dockerizing a set of packages based on private repositories on Travis. $\endgroup$
    – I'll Eat My Hat
    Commented Jul 22, 2020 at 18:20
13
$\begingroup$

First export environment configuration of your current conda environment using:

conda-env  export -n your_env_name > your_env_name.yml

example:

conda-env  export -n base> base.yml

After running above command their should be yml configuration file in your current directory which contain information of your conda environment

To create new environment using yml configuration file run:

conda-env create -n new_env -f=\path\to\base.yml

example:

conda-env create -n venv -f=base.yml

In case the above one does not work (due to the various issues of conda itself), it's always worth a try with the following variation:

conda-env create --name new_env --file \path\to\base.yml
Improve this answer
$\endgroup$
1
  • $\begingroup$ In my conda, the syntax is conda env export, not conda-env export $\endgroup$
    – Christian Seitz
    Commented Sep 12, 2023 at 17:52
8
$\begingroup$

If your program is mostly Python, you could rely solely on virtual environments.

Create virtual environments to isolate your dependencies rather than using the system libraries. Then use virtual environment tools to duplicate your environments.

In the working virtualenv, create a file with the version of each installed Python library :

pip freeze > requirements.txt

In the new virtualenv, ask pip to install those libraries with the same version :

pip install -r requirements.txt

This makes sure you get the same lib versions on both machines. And since requirements.txt is tracked by your VCS, you can always recreate the environment of an old version of your code.

Of course, if you need a database, a production web server, etc. you end up with a few more steps and you can't rely on virtualenv to ensure both environments match. This is where Docker steps in (See Pieter21's answer).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I didn't notice the anaconda tag on your question. I'm not experienced with this, but be careful. I think anaconda has it's own way of manging environments and using both anaconda and virtualenv could get you into trouble. However, I suppose anaconda should offer equivalent features. $\endgroup$
    – Jérôme
    Commented Oct 26, 2017 at 13:12
  • $\begingroup$ Great help for people who use pip instead of conda. $\endgroup$
    – tomjpsun
    Commented Aug 14, 2020 at 9:41
6
$\begingroup$

Look into 'containers', e.g. Docker (https://www.docker.com/what-container), a more lightweight alternative to virtualization.

It will require some time investment but in the end will provide many benefits.

From the link, where I marked your specific need in bold italic:

Package software into standardized units for development, shipment and deployment

A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. Available for both Linux and Windows based apps, containerized software will always run the same, regardless of the environment. Containers isolate software from its surroundings, for example differences between development and staging environments and help reduce conflicts between teams running different software on the same infrastructure.

Improve this answer
$\endgroup$
1
  • $\begingroup$ even with docker, you can find yourself with multiple conda envs within $\endgroup$
    – Monica Heddneck
    Commented Jun 12, 2020 at 20:40
4
$\begingroup$

A wrap up of the existing ways to create an environment based on another one:

  • Cloning an environment:

    • From an existing environment:

      $ conda create --name NEW_ENV_NAME --clone ORIG_ENV_NAME

    • From an exported environment file on the same machine:

      $ conda create --name ENV_NAME —-file FILE_NAME.yml

    • From an exported environment file on a different machine:

    $ conda env export > ENV_NAME.yml
$ conda env create -f ENV_NAME.yml
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ $ conda create --name NEW_ENV_NAME --clone ORIG_ENV_NAME $\endgroup$
    – B. Sun
    Commented Aug 26, 2019 at 2:35
3
$\begingroup$

From the very end of this documentation page:

Save packages for future use:

conda list --export > package-list.txt

Reinstall packages from an export file:

conda create -n myenv --file package-list.txt
Improve this answer
$\endgroup$
2
$\begingroup$

One-liner:

conda create --clone source_env --name destination_env
Improve this answer
$\endgroup$
1
$\begingroup$

to clone a miniconda/anaconda environment for Windows users:

  • install miniconda/anaconda with "all users" option. if you install with your current user, the path will be set during the compile of the exe files and you are lost

  • using conda-pack is not bad but for me zipping the complete ~/<anaconda/miniconda> path and unzipping it on the target machine was working properly too.

btw. do not forget to copy the dlls libcrypto-xxx.dll and libssl-xxx.dll from the folder ~/Library/bin to the folder ~/DLLs

Improve this answer
$\endgroup$
0
$\begingroup$

TLDR: the best method is below:

first, in old server

rsync -avr /home/xxxx/anaconda3/envs/your_env_name your_login_name@new_server_ip:/home/xxxx/miniconda3/envs/

then in new server conda activate your_env_name, and run pip list, you'll get error: /bin/python: bad interpreter: No such file or directory： bad interpreter: No such file or directory

second, in new server

find ~/miniconda3/envs/nn/ -type f  -exec grep --binary-files=without-match -rl "oldprefix" {} \; -exec sed -i 's+newprefix+g' {} +

the below replace all python prefix in envs dir, make commandline command like pip, work as expect.

--------------------------------separating line-----------------------------------

if you dont want to redownload packages in the new server (of same os platform), the official method is use pip install conda-pack . conda-pack archives an entire environment including the package binaries, which is useful in low or no bandwidth situations.

Ab oberversation, by cat $(which pip) you can find the pip command is just a linux file(with execute privilige assisiation to a python.exe path):

#!/home/your_home_name/miniconda3/envs/your_env_name/bin/python

# -*- coding: utf-8 -*-
import re
import sys

from pip._internal.cli.main import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])

see the first line with #! shellbang, all python module command realized by entry_point such as pip, need to replace this python path prefix when move to new server.

AFAIK, what conda pack do is replace all prefix in old envs file form #!/home/your_home_name/miniconda3/envs/your_env_name/bin/python to #!/usr/bin/python, and conda unpack replace it to another path in your new server's conda directory

howerver with conda pack you need to tar.gzip first and rsync to new server, then untargzip.

the best method is below:

first, in old server

rsync -avr /home/xxxx/anaconda3/envs/your_env_name your_login_name@new_server_ip:/home/xxxx/miniconda3/envs/

then in new server conda activate your_env_name, and run pip list, you'll get error: /bin/python: bad interpreter: No such file or directory： bad interpreter: No such file or directory

second, in new server

find ~/miniconda3/envs/nn/ -type f  -exec grep --binary-files=without-match -rl "oldprefix" {} \; -exec sed -i 's+newprefix+g' {} +

the below replace all python prefix in envs dir, make commandline command like pip, work as expect.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.