26
$\begingroup$

I developed a machine learning model with Python (Anaconda + Flask) on my workstation and all goes well. Later, I tried to ship this program onto another machine where of course I tried to set up the same environment, but the program fails to run. I copied the program to other machines where it also runs smoothly.

I cannot figure out what the problem is in the failed case (both the program code and the error message are copious so I am not able to present them here) but I'm almost certain that it is something with the different versions of the dependencies.

So, my question is that given an environment where a certain program runs well, how can I clone it to another where it should run well also? Of course, without the cloning of the full system ;)

$\endgroup$
  • $\begingroup$ Use conda env export. $\endgroup$ – kbrose Oct 26 '17 at 14:04
37
$\begingroup$

First of all this is a Python/Anaconda question and should probably be asked in a different stack exchange subsite.


As for the question itself - you can export your Anaconda environment using:

conda env export > environment.yml

And recreate it using:

conda env create -f environment.yml

Please note that as others suggested - you should use virtual environments which allows you to create a certain environment that is separated from that of your machine and manage it more easily.

To create a virtual environment in Anaconda you can use:

conda create -n yourenvname python=x.x anaconda

which you activate using:

source activate yourenvname
$\endgroup$
  • $\begingroup$ The Anaconda documentation is not very clear on wether to use conda create or conda env create when sharing/recreating an environment. Could you please further detail why you recommend using conda env create in this situation? $\endgroup$ – Tanguy Nov 15 '18 at 10:05
  • $\begingroup$ You can find some notes about the difference between conda create and conda env create here: groups.google.com/a/continuum.io/forum/#!topic/conda/… That said, I think you could usually use them interchangeably. $\endgroup$ – ginge Nov 15 '18 at 13:21
  • 1
    $\begingroup$ I have seen this thread, but precisely I am trying to understand precisely in which situation each option (conda create vs conda env create) should be prefered and what are the downsides of each (e.g. : "[conda env create is for] environments in which packages using pip have been installed into, which causes additional complexity" : what kind of additional complexity does it add?). $\endgroup$ – Tanguy Nov 15 '18 at 13:52
  • $\begingroup$ I wanna ask that when running conda env create -f environment.yml, this will cause error because the name of virtenv in yml file has already been used. Change the name to your new virtenv to overcome. $\endgroup$ – Giang Nguyễn Apr 21 at 9:10
  • $\begingroup$ This way saves your life! if you create an environment from a file based on Anaconda Managing Environments page instructions, doesn't work if you use another platform. conda list --explicit > FILE_NAME exports binaries for the current platform and apparently not working on another one. $\endgroup$ – Shayan Amani Jul 12 at 20:33
5
$\begingroup$

Look into 'containers', e.g. Docker (https://www.docker.com/what-container), a more lightweight alternative to virtualization.

It will require some time investment but in the end will provide many benefits.

From the link, where I marked your specific need in bold italic:

Package software into standardized units for development, shipment and deployment

A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings. Available for both Linux and Windows based apps, containerized software will always run the same, regardless of the environment. Containers isolate software from its surroundings, for example differences between development and staging environments and help reduce conflicts between teams running different software on the same infrastructure.

$\endgroup$
4
$\begingroup$

If your program is mostly Python, you could rely solely on virtual environments.

Create virtual environments to isolate your dependencies rather than using the system libraries. Then use virtual environment tools to duplicate your environments.

In the working virtualenv, create a file with the version of each installed Python library :

pip freeze > requirements.txt

In the new virtualenv, ask pip to install those libraries with the same version :

pip install -r requirements.txt

This makes sure you get the same lib versions on both machines. And since requirements.txt is tracked by your VCS, you can always recreate the environment of an old version of your code.

Of course, if you need a database, a production web server, etc. you end up with a few more steps and you can't rely on virtualenv to ensure both environments match. This is where Docker steps in (See Pieter21's answer).

$\endgroup$
  • $\begingroup$ I didn't notice the anaconda tag on your question. I'm not experienced with this, but be careful. I think anaconda has it's own way of manging environments and using both anaconda and virtualenv could get you into trouble. However, I suppose anaconda should offer equivalent features. $\endgroup$ – Jérôme Oct 26 '17 at 13:12
4
$\begingroup$

First export environment configuration of your current conda environment using:

conda-env  export -n your_env_name > your_env_name.yml

example:

conda-env  export -n base> base.yml

After running above command their should be yml configuration file in your current directory which contain information of your conda environment

To create new environment using yml configuration file run:

conda-env create -n new_env -f=\path\to\base.yml 

example:

conda-env create -n venv -f=base.yml

In case the above one does not work (due to the various issues of conda itself), it's always worth a try with the following variation:

conda-env create --name new_env --file \path\to\base.yml 
$\endgroup$
1
$\begingroup$

From the very end of this documentation page:

Save packages for future use:

conda list --export > package-list.txt

Reinstall packages from an export file:

conda create -n myenv --file package-list.txt
$\endgroup$
1
$\begingroup$

A wrap up of the existing ways to create an environment based on another one:

  • Cloning an environment:

    • From an existing environment:

      $ conda create --name ORIG_ENV_NAME --clone CLONE_ENV_NAME

    • From an exported environment file on the same machine:

      $ conda create --name ENV_NAME —-file FILE_NAME.yml

    • From an exported environment file on a different machine:
      $ conda env export > ENV_NAME.yml
      $ conda env create -f ENV_NAME.yml```
      
$\endgroup$
  • $\begingroup$ $ conda create --name NEW_ENV_NAME --clone ORIG_ENV_NAME $\endgroup$ – B. Sun Aug 26 at 2:35
0
$\begingroup$

One-liner

conda create --clone source_env --name destination_env

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.