Vagrant environment setup tutorial¶
This section guides first-time contributors through installing the Zulip development environment on Windows, macOS, and Ubuntu.
The recommended method for installing the Zulip development environment is to use Vagrant with VirtualBox on Windows and macOS, and Vagrant with LXC on Ubuntu. This method creates a virtual machine (for Windows and macOS) or a Linux container (for Ubuntu) inside which the Zulip server and all related services will run.
- Step 1: Install Prerequisites
- Step 2: Get Zulip code
- Step 3: Start the development environment
- Step 4: Developing
- Troubleshooting & Common Errors
- Specifying a proxy
If you encounter errors installing the Zulip development environment, check Troubleshooting & Common Errors. If that doesn’t help, please visit #provision help in the Zulip development community server for real-time help, send a note to the Zulip-devel Google group or file an issue.
When reporting your issue, please include the following information:
- host operating system
- installation method (Vagrant or direct)
- whether or not you are using a proxy
- a copy of Zulip’s
vagrantprovisioning logs, available in
/var/log/provision.logon your virtual machine
Installing the Zulip development environment requires downloading several hundred megabytes of dependencies. You will need an active internet connection throughout the entire installation processes. (See Specifying a proxy if you need a proxy to access the internet.)
- All: 2GB available RAM, Active broadband internet connection.
- macOS: macOS (10.11 El Capitan or 10.12 Sierra recommended), Git, VirtualBox, Vagrant.
- Ubuntu: 14.04 64-bit or 16.04 64-bit, Git, Vagrant, lxc.
- Windows: Windows 64-bit (Win 10 recommended), hardware virtualization enabled (VT-X or AMD-V), administrator access, Git for Windows (which installs Git BASH), VirtualBox, Vagrant.
Don’t see your system listed above? See Advanced setup for details about installing for other Linux and UNIX platforms.
Step 1: Install Prerequisites¶
Now you are ready for Step 2: Get Zulip Code..
The setup for Ubuntu 14.04 Trusty and Ubuntu 16.04 Xenial are the same.
If you’re in a hurry, you can copy and paste the following into your terminal after which you can jump to Step 2: Get Zulip Code:
sudo apt-get -y purge vagrant && \ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb && \ sudo dpkg -i vagrant*.deb && \ sudo apt-get -y install build-essential git ruby lxc lxc-templates cgroup-lite redir && \ vagrant plugin install vagrant-lxc && \ vagrant lxc sudoers
For a step-by-step explanation, read on.
1. Install Vagrant¶
For both 14.04 Trusty and 16.04 Xenial, you’ll need a more recent version of Vagrant than what’s available in the official Ubuntu repositories.
First uninstall any vagrant package you may have installed from the Ubuntu repository:
christie@ubuntu-desktop:~ $ sudo apt-get purge vagrant
Now download and install the .deb package for Vagrant 1.8.6:
christie@ubuntu-desktop:~ $ wget https://releases.hashicorp.com/vagrant/1.8.6/vagrant_1.8.6_x86_64.deb christie@ubuntu-desktop:~ $ sudo dpkg -i vagrant*.deb
2. Install remaining dependencies¶
Now install git and lxc-related packages:
christie@ubuntu-desktop:~ $ sudo apt-get install build-essential git ruby lxc lxc-templates cgroup-lite redir
3. Install the vagrant lxc plugin:¶
christie@ubuntu-desktop:~ $ vagrant plugin install vagrant-lxc Installing the 'vagrant-lxc' plugin. This can take a few minutes... Installed the plugin 'vagrant-lxc (1.2.1)'!
If you encounter an error when trying to install the vagrant-lxc plugin, see this.
4. Configure sudo to be passwordless¶
christie@ubuntu-desktop:~ $ vagrant lxc sudoers [sudo] password for christie:
If you encounter an error running
vagrant lxc sudoers, see
Now you are ready for Step 2: Get Zulip Code.
- Install Git for Windows, which installs Git BASH.
- Install VirtualBox (version >= 5.1.6).
- Install Vagrant (version 1.8.4-1.8.6, do not use 1.8.7).
(Note: While Git BASH is recommended, you may also use Cygwin. If you do, make sure to install default required packages along with git, curl, openssh, and rsync binaries.)
After installing, you must run Git BASH as an administrator.
Also, you must have hardware virtualization enabled (VT-X or AMD-V) in your computer’s BIOS.
Step 2: Get Zulip Code¶
If you haven’t already created an ssh key and added it to your GitHub account, you should do that now by following these instructions.
- In your browser, visit https://github.com/zulip/zulip
and click the
forkbutton. You will need to be logged in to GitHub to do this.
- Open Terminal (macOS/Ubuntu) or Git BASH (Windows; must run as an Administrator).
- In Terminal/Git BASH, clone your fork:
git clone firstname.lastname@example.org:YOURUSERNAME/zulip.git
This will create a ‘zulip’ directory and download the Zulip code into it.
Don’t forget to replace YOURUSERNAME with your git username. You will see something like:
christie@win10 ~ $ git clone email@example.com:YOURUSERNAME/zulip.git Cloning into 'zulip'... remote: Counting objects: 73571, done. remote: Compressing objects: 100% (2/2), done. remote: Total 73571 (delta 1), reused 0 (delta 0), pack-reused 73569 Receiving objects: 100% (73571/73571), 105.30 MiB | 6.46 MiB/s, done. Resolving deltas: 100% (51448/51448), done. Checking connectivity... done. Checking out files: 100% (1912/1912), done.`
Now you are ready for Step 3: Start the development environment.
Step 3: Start the development environment¶
Change into the zulip directory and tell vagrant to start the Zulip
development environment with
christie@win10 ~ $ cd zulip christie@win10 ~/zulip $ vagrant up
The first time you run this command it will take some time because vagrant does the following:
- downloads the base Ubuntu 14.04 virtual machine image (for macOS and Windows) or container (for Ubuntu)
- configures this virtual machine/container for use with Zulip,
- creates a shared directory mapping your clone of the Zulip code inside the
virtual machine/container at
- runs the
tools/provisionscript inside the virtual machine/container, which downloads all required dependencies, sets up the python environment for the Zulip development server, and initializes a default test database. We call this process “provisioning”.
You will need an active internet connection during the entire
process. (See Specifying a proxy if you need a
proxy to access the internet.)
vagrant up can fail while
provisioning if your Internet connection is unreliable. To retry, you
vagrant provision (
vagrant up will just boot the guest
without provisioning after the first time). Other common issues are
documented in the
Troubleshooting & Common Errors
section. If that doesn’t help, please visit
in the Zulip development community server for
On Windows, you will see
The system cannot find the path specified. message
several times. This is expected behavior and is not an error.
vagrant up has completed, connect to the development
christie@win10 ~/zulip $ vagrant ssh
You should see something like this on Windows and macOS:
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64) * Documentation: https://help.ubuntu.com/ System information as of Wed May 4 21:45:43 UTC 2016 System load: 0.61 Processes: 88 Usage of /: 3.5% of 39.34GB Users logged in: 0 Memory usage: 7% IP address for eth0: 10.0.2.15 Swap usage: 0% Graph this data and manage this system at: https://landscape.canonical.com/ Get cloud support with Ubuntu Advantage Cloud Guest: http://www.ubuntu.com/business/services/cloud 0 packages can be updated. 0 updates are security updates.
Or something as brief as this in the case of Ubuntu:
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 4.4.0-21-generic x86_64) * Documentation: https://help.ubuntu.com/
Congrats, you’re now inside the Zulip development environment!
You can confirm this by looking at the command prompt, which starts
(zulip-venv)vagrant@. If it just starts with
provisioning failed and you should look at the
Next, start the Zulip server:
(zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ cd zulip (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip $ ./tools/run-dev.py
You will see several lines of output starting with something like:
2016-05-04 22:20:33,895 INFO: process_fts_updates starting Recompiling templates 2016-05-04 18:20:34,804 INFO: Not in recovery; listening for FTS updates done Validating Django models.py... System check identified no issues (0 silenced). Django version 1.8 Tornado server is running at http://localhost:9993/ Quit the server with CTRL-C. 2016-05-04 18:20:40,716 INFO Tornado loaded 0 event queues in 0.001s 2016-05-04 18:20:40,722 INFO Tornado 95.5% busy over the past 0.0 seconds Performing system checks...
And ending with something similar to:
http://localhost:9994/webpack-dev-server/ webpack result is served from http://localhost:9991/webpack/ content is served from /srv/zulip webpack: bundle is now VALID. 2016-05-06 21:43:29,553 INFO Tornado 31.6% busy over the past 10.6 seconds 2016-05-06 21:43:35,007 INFO Tornado 23.9% busy over the past 16.0 seconds
Now the Zulip server should be running and accessible. Verify this by navigating to http://localhost:9991/ in the browser on your main machine.
You should see something like this:
The Zulip server will continue to run and send output to the terminal window. When you navigate to Zulip in your browser, check your terminal and you should see something like:
2016-05-04 18:21:57,547 INFO 127.0.0.1 GET 302 582ms (+start: 417ms) / (unauth via ?) [04/May/2016 18:21:57]"GET / HTTP/1.0" 302 0 2016-05-04 18:21:57,568 INFO 127.0.0.1 GET 301 4ms /login (unauth via ?) [04/May/2016 18:21:57]"GET /login HTTP/1.0" 301 0 2016-05-04 18:21:57,819 INFO 127.0.0.1 GET 200 209ms (db: 7ms/2q) /login/ (unauth via ?)
Now you’re ready for Step 4: Developing.
Step 4: Developing¶
Where to edit files¶
You’ll work by editing files on your host machine, in the directory where you cloned Zulip. Use your favorite editor (Sublime, Atom, Vim, Emacs, Notepad++, etc.).
When you save changes they will be synced automatically to the Zulip development environment on the virtual machine/container.
Each component of the Zulip development server will automatically restart itself or reload data appropriately when you make changes. So, to see your changes, all you usually have to do is reload your browser. More details on how this works are available below.
Don’t forget to read through the code style guidelines for details about how to configure your editor for Zulip. For example, indentation should be set to 4 spaces rather than tabs.
Understanding run-dev.py debugging output¶
It’s good to have the terminal running
run-dev.py up as you work since error
messages including tracebacks along with every backend request will be printed
See Logging for further details on the run-dev.py console output.
Committing and pushing changes with git¶
When you’re ready to commit or push changes via git, you will do this by running git commands in Terminal (macOS/Ubuntu) or Git BASH (Windows) in the directory where you cloned Zulip on your main machine.
If you’re new to working with Git/GitHub, check out our Git & GitHub Guide.
Maintaining the development environment¶
If after rebasing onto a new version of the Zulip server, you receive
new errors while starting the Zulip server or running tests, this is
probably not because Zulip’s master branch is broken. Instead, this
is likely because we’ve recently merged changes to the development
environment provisioning process that you need to apply to your
development environment. To update your environment, you’ll need to
re-provision your vagrant machine using
vagrant provision (this just
tools/provision from your Zulip checkout inside the Vagrant
guest); this should complete in about a minute.
After provisioning, you’ll want to (re)start the Zulip development server.
Rebuilding the development environment¶
If you ever want to recreate your development environment again from
scratch (e.g. to test as change you’ve made to the provisioning
process, or because you think something is broken), you can do so
vagrant destroy and then
vagrant up. This will usually be
much faster than the original
vagrant up since the base image is
already cached on your machine (it takes about 5 minutes to run with a
fast Internet connection).
Any additional programs (e.g. Zsh, emacs, etc.) or configuration that
you may have installed in the development environment will be lost
when you recreate it. To address this, you can create a script called
tools/custom_provision in your Zulip Git checkout; and place any
extra setup commands there. Vagrant will run
every time you run
vagrant provision (or create a Vagrant guest via
Shutting down the development environment for use later¶
To shut down but preserve the development environment so you can use
it again later use
vagrant halt or
You can do this from the same Terminal/Git BASH window that is running
run-dev.py by pressing ^C to halt the server and then typing
exit. Or you
can halt vagrant from another Terminal/Git BASH window.
From the window where run-dev.py is running:
2016-05-04 18:33:13,330 INFO 127.0.0.1 GET 200 92ms /register/ (unauth via ?) ^C KeyboardInterrupt (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip$ exit logout Connection to 127.0.0.1 closed. christie@win10 ~/zulip
Now you can suspend the development environment:
christie@win10 ~/zulip $ vagrant suspend ==> default: Saving VM state and suspending execution...
vagrant suspend doesn’t work, try
christie@win10 ~/zulip $ vagrant halt ==> default: Attempting graceful shutdown of VM...
Resuming the development environment¶
When you’re ready to work on Zulip again, run
vagrant up. You will also need
to connect to the virtual machine with
vagrant ssh and re-start the Zulip
christie@win10 ~/zulip $ vagrant up $ vagrant ssh (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ cd zulip (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip $ ./tools/run-dev.py
Next, read the following to learn more about developing for Zulip:
Troubleshooting & Common Errors¶
Below you’ll find a list of common errors and their solutions. Most
issues are resolved by just provisioning again (via
tools/provision.py inside the Vagrant guest or equivalently
vagrant provision from outside).
If these solutions aren’t working for you or you encounter an issue not documented below, there are a few ways to get further help:
- Ask in #provision help in the Zulip development community server,
- send a note to the Zulip-devel Google group, or
- File an issue.
When reporting your issue, please include the following information:
- host operating system
- installation method (Vagrant or direct)
- whether or not you are using a proxy
- a copy of Zulip’s
vagrantprovisioning logs, available in
/var/log/provision.logon your virtual machine. If you choose to post just the error output, please include the beginning of the error output, not just the last few lines.
The output of
tools/diagnose run inside the Vagrant guest is also
Vagrant guest doesn’t show (zulip-venv) at start of prompt¶
This is caused by provisioning failing to complete successfully. You
can see the errors in
var/log/provision.log; it should end with
something like this:
ESC[94mZulip development environment setup succeeded!ESC[0m
ESC stuff are the terminal color codes that make it show as a nice
blue in the terminal, which unfortunately looks ugly in the logs.
If you encounter an incomplete
/var/log/provision.log file, you need to
update your environment. Re-provision your vagrant machine; if the problem
persists, please come chat with us (see instructions above) for help.
After you provision successfully, you’ll need to exit your
shell and run
vagrant ssh again to get the virtualenv setup properly.
The box ‘ubuntu/trusty64’ could not be found¶
If you see the following error when you run
The box 'ubuntu/trusty64' could not be found or could not be accessed in the remote catalog. If this is a private box on HashiCorp's Atlas, please verify you're logged in via `vagrant login`. Also, please double-check the name. The expanded URL and error message are shown below: URL: ["https://atlas.hashicorp.com/ubuntu/trusty64"]
Then the version of
curl that ships with Vagrant is not working on your
machine. You are most likely to encounter this error on Windows/Cygwin and
On macOS this error is most likely to occur with Vagrant version 1.8.7 and is a known issue.
The solution is to downgrade Vagrant to version 1.8.6 (available
here), or to use your system’s
curl instead of the one that ships with Vagrant:
sudo ln -nsf /usr/bin/curl /opt/vagrant/embedded/bin/curl
On Windows/Cygwin, the fix is simple: replace it with the version from Cygwin.
First, determine the location of Cygwin’s curl with
christie@win10 ~/zulip $ which curl /usr/bin/curl
Now determine the location of Vagrant with
christie@win10 ~/zulip $ which vagrant /cygdrive/c/HashiCorp/Vagrant/bin/vagrant
The path up until
/bin/vagrant is what you need to know. In the example above it’s
Finally, copy Cygwin’s curl to Vagrant
christie@win10 ~/zulip $ cp /usr/bin/curl.exe /cygdrive/c/HashiCorp/Vagrant/embedded/bin/
vagrant up and vagrant should be able to fetch the required
Connection timeout on
If you see the following error after running
default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying... default: Error: Connection timeout. Retrying...
A likely cause is that hardware virtualization is not enabled for your computer. This must be done via your computer’s BIOS settings. Look for a setting called VT-x (Intel) or (AMD-V).
If this is already enabled in your BIOS, double-check that you are running a 64-bit operating system.
For further information about troubleshooting vagrant timeout errors see this post.
Vagrant up fails with subprocess.CalledProcessError¶
vagrant up command basically does the following:
- Downloads an Ubuntu image and starts it using a Vagrant provider.
vagrant sshto connect to that Ubuntu guest, and then runs
tools/provision, which has a lot of subcommands that are executed via Python’s
subprocessmodule. These errors mean that one of those subcommands failed.
To debug such errors, you can log in to the Vagrant guest machine by
vagrant ssh, which should present you with a standard shell
prompt. You can debug interactively by using e.g.
cd zulip && ./tools/provision, and then running the individual subcommands
that failed. Once you’ve resolved the problem, you can rerun
tools/provision to proceed; the provisioning system is designed
to recover well from failures.
The zulip provisioning system is generally highly reliable; the most common cause of issues here is a poor network connection (or one where you need a proxy to access the Internet and haven’t configured the development environment to use it.
Once you’ve provisioned successfully, you’ll get output like this:
Zulip development environment setup succeeded! (zulip-venv) vagrant@vagrant-base-trusty-amd64:~/zulip$
(zulip-venv) part is missing, this is because your
installation failed the first time before the Zulip virtualenv was
created. You can fix this by just closing the shell and running
vagrant ssh again, or using
Finally, if you encounter any issues that weren’t caused by your Internet connection, please report them! We try hard to keep Zulip development environment provisioning free of bugs.
pip install fails during
vagrant up on Ubuntu¶
Likely causes are:
- Networking issues
- Insufficient RAM. Check whether you’ve allotted at least two gigabytes of RAM, which is the minimum Zulip requires. If not, go to your VM settings and increase the RAM, then restart the VM.
npm install errors¶
tools/provision script may encounter an error related to
that looks something like:
==> default: + npm install ==> default: Traceback (most recent call last): ==> default: File "/srv/zulip/tools/provision", line 195, in <module> ==> default: ==> default: sys.exit(main()) ==> default: File "/srv/zulip/tools/provision", line 191, in main ==> default: ==> default: run(["npm", "install"]) ==> default: File "/srv/zulip/scripts/lib/zulip_tools.py", line 78, in run ==> default: ==> default: raise subprocess.CalledProcessError(rc, args) ==> default: subprocess ==> default: . ==> default: CalledProcessError ==> default: : ==> default: Command '['npm', 'install']' returned non-zero exit status 34 The SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed. The output for this command should be in the log above. Please read the output to determine what went wrong.
Usually this error is not fatal. Try connecting to the development environment and re-trying the command from withing the virtual machine:
christie@win10 ~/zulip $ vagrant ssh (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~ $ cd zulip (zulip-venv)vagrant@vagrant-ubuntu-trusty-64:~/zulip $ npm install npm WARN optional Skipping failed optional dependency /chokidar/fsevents: npm WARN notsup Not compatible with your operating system or architecture: firstname.lastname@example.org
These are just warnings so it is okay to proceed and start the Zulip server.
When building the development environment using Vagrant and the LXC provider,
if you encounter permissions errors, you may need to
chown -R 1000:$(whoami) /path/to/zulip on the host before running
vagrant up in order to ensure that
the synced directory has the correct owner during provision. This issue will
arise if you run
id username on the host where
username is the user running
Vagrant and the output is anything but 1000. This seems to be caused by
Vagrant behavior; for more information, see the vagrant-lxc FAQ entry about
shared folder permissions.
If you see the following error when you try to install the vagrant-lxc plugin:
/usr/lib/ruby/2.3.0/rubygems/specification.rb:946:in `all=': undefined method `group_by' for nil:NilClass (NoMethodError) from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:275:in `with_isolated_gem' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:231:in `internal_install' from /usr/lib/ruby/vendor_ruby/vagrant/bundler.rb:102:in `install' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:62:in `block in install_plugin' from /usr/lib/ruby/vendor_ruby/vagrant/plugin/manager.rb:72:in `install_plugin' from /usr/share/vagrant/plugins/commands/plugin/action/install_gem.rb:37:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/warden.rb:34:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/builder.rb:116:in `call' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `block in run' from /usr/lib/ruby/vendor_ruby/vagrant/util/busy.rb:19:in `busy' from /usr/lib/ruby/vendor_ruby/vagrant/action/runner.rb:66:in `run' from /usr/share/vagrant/plugins/commands/plugin/command/base.rb:14:in `action' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:32:in `block in execute' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `each' from /usr/share/vagrant/plugins/commands/plugin/command/install.rb:31:in `execute' from /usr/share/vagrant/plugins/commands/plugin/command/root.rb:56:in `execute' from /usr/lib/ruby/vendor_ruby/vagrant/cli.rb:42:in `execute' from /usr/lib/ruby/vendor_ruby/vagrant/environment.rb:268:in `cli' from /usr/bin/vagrant:173:in `<main>'
And you have vagrant version 1.8.1, then you need to patch vagrant manually. See this post for an explanation of the issue, which should be fixed when Vagrant 1.8.2 is released.
In the meantime, read this post for how to create and apply the patch.
It will look something like this:
christie@xenial:~ $ sudo patch --directory /usr/lib/ruby/vendor_ruby/vagrant < vagrant-plugin.patch patching file bundler.rb
Specifying a proxy¶
If you need to use a proxy server to access the Internet, you will
need to specify the proxy settings before running
First, install the Vagrant plugin
vagrant plugin install vagrant-proxyconf.
~/.zulip-vagrant-config and add the following lines to
it (with the appropriate values in it for your proxy):
HTTP_PROXY http://proxy_host:port HTTPS_PROXY http://proxy_host:port NO_PROXY localhost,127.0.0.1,.example.com
vagrant up in your terminal to install the development
server. If you ran
vagrant up before and failed, you’ll need to run
vagrant destroy first to clean up the failed installation.
You can also change the port on the host machine that Vagrant uses by
adding to your
~/.zulip-vagrant-config file. E.g. if you set:
(and halt and restart the Vagrant guest), then you would visit http://localhost:9971/ to connect to your development server.
If you’d like to be able to connect to your development environment from other machines than the VM host, you can manually set the host IP address in the ‘~/.zulip-vagrant-config’ file as well. For example, if you set:
(and restart the Vagrant guest), your host IP would be 0.0.0.0, a special value for the IP address that means any IP address can connect to your development server.