Deeden

• Articles
• Reading
• Podcast
• Feeds

Migrating ExpressionEngine from development to production

25th of June, 2008 | 25 comments

Moving an ExpressionEngine site from one server to another, or even one domain to another domain, is a complicated process. Whenever I’ve attempted to do this I’ve found that the hardest part is making sure that the database is correct for the new location. This involves copying the database across and then making sure I change every path and domain through the control panel to the new values.

I’ve never succeeded in migrating successfully using the manual method. I always miss something and have to deal with some small problem that inevitably crops up. Finally, I decided to make it easier on myself, so I wrote a script to perform the update for me. This will not be for everyone, however if you’re comfortable with command line scripts and ruby this may do the job for you.

The script (the impatient among you can find it at the end of this article) does one job and one job only. It takes a MySQL dump from an EE database and makes the updates required for any domain name and path changes, writing these out again in the format of another MySQL dump. For example, if I tell it that my development site lives at /www/deeden.tld and my production site at /www/deeden.co.uk it updates the database to reflect this, dealing with all of the complications that arise, such as correctly dealing with the serialisation which some of the tables use.

An example of performing the above update would be…

ee_migrator --current_domain=deeden.tld --new_domain=deeden.co.uk --current_path=/Users/steve/Sites/deeden.tld --new_path=/www/deeden.co.uk development_sql.txt > live_sql.txt

This runs the script (assuming it is named ee_migrator) and changes all occurrences of the domain deeden.tld to deeden.co.uk as well as changing the path to the web site from /Users/steve/Sites/deeden.tld to /www/deeden.co.uk. Rather useful when I’ve been developing a site on my iBook and want to move it to the live server.

I’ll put together a page with full instructions on the use of the script in the next few days. For now I’ll give a brief explanation of the parameters…

-h, --help

Display usage instructions.

-t, --table_prefix

Specify a different table prefix from the default, which is exp.

-a, --all_data

Keep all data from the initial dump. By default the script ignores a number of tables which should not be copied from a development, such as control panel logs and email caches. The --all_data allows all data to be copied, unsurprisingly.

-c, --current_domain <OLD DOMAIN>

The domain being moved from. In the example above it would be deeden.tld. In the case where the domain isn’t changing simply leave this option out.

-n, --new_domain <NEW DOMAIN>

The domain being moved to. In the example above it would be deeden.co.uk. In the case where the domain isn’t changing simply leave this option out as well.

-s, --current_path <CURRENT SITE PATH>

The path to where the old domain lives. If the path isn’t being changed then leave the option out.

-p, --new_path <NEW SITE PATH>

Specify the new path to the web directory. Again, if this isn’t being changed simply leave it out.

The input file is a mysqldump file, and the output is the same file with the appropriate changes made. Generally I do a dump of my development database, do a quick scan of the output to see that everything looks alright and them import the new database file into my a fresh database for the live site. Your mileage might vary.

There are a few things to bear in mind if you do plan to use this script, such as…

• Backup your database.
• I accept no responsibility for any damage this script may do. Use it at your own risk.
• If you find a bug please do report it.
• If you make an enhancement please send it to me, it may be useful to other people.
• Seriously, backup your database.

The most recent version of the script is version 1.4. You can also find the script on github.

25 comments

Kris Khaira / 24th of October, 2008 / #1

Just to let everyone know, I’ve been using this method for several projects already without any problems. Thanks to Stephen Rushe for making my work easier!

Stephen / 8th of December, 2008 / #2

Kris, I’m glad it has been of use :)

Matthew Spiel / 30th of January, 2009 / #3

I just used the script on my site after I moved to my new server… it is awesome!

but the script didn’t work (because my url was a little funky [sub domain and a /js on the end of it)... what I found that most places don’t say (EE wiki, etc) is that you need to go into your Cache folder (system/cache) and nuc (nook!) everything that is in there… this will allow the DB and EE to rewrite them with the proper permissions…

Stephen / 30th of January, 2009 / #4

The sub-domain shouldn’t be an issue Matthew. I’ve used it on sub-domains before without any problems. Feel free to tell me the paths you used, I’ll have a look and see if they should cause any trouble.

As for the cache thing, that’s outside the remit of the script. I deliberately wrote it to do the one job of updating the database :)

Wes Baker / 19th of March, 2009 / #5

I appreciate the work you’ve done here, but I ran the script and nothing seemed to have changed.

You can see the original file here: old.sql
And the new file here: new.sql

Wes Baker / 19th of March, 2009 / #6

I suppose it would help to see the command:

ruby ee_migrator.rb -c eeorig -n ee -s /Users/wes/Sites/ee -p /Users/wes/Sites/ee2 expressionengine.sql > newsql.sql

Stephen / 19th of March, 2009 / #7

Thanks for the report Wes. I’ve fixed the issue (case-sensitivity) and you can get a new copy of the script from the same place as before.

Wes Baker / 19th of March, 2009 / #8

Thanks for the quick turn around, after trying it out I’m getting this error:

The option—current_path must have a leading slash
ee_migrator.rb:93: undefined method `<<’ for nil:NilClass (NoMethodError)
  from ee_migrator.rb:88:in `each’
  from ee_migrator.rb:88


Here’s the command I’m running:

ruby ee_migrator.rb -c http://eeorig/ -n http://ee/ expressionengine.sql < newsite.sql


Stephen / 19th of March, 2009 / #9

What will I do with you? ;)

I’ll have a look at it now. The football is on and I can sit and happily look at this while listening to that.

Stephen / 19th of March, 2009 / #10

Wes, I’ve put a new version here for you to try out. It should fix the issue you reported. I’ve also extended it to allow http:// in the domain parameters.

Let me know how it goes. If it’s fine I’ll replace the existing version with it.

Wes Baker / 20th of March, 2009 / #11

Stephen,

It mostly worked. Using the same command and files, it replaced all instances of eeorig with ee. However, it seemed to delete some of the inserts. You can see what happened in the files I posted earlier—I updated them.

Stephen / 20th of March, 2009 / #12

By default the script ignores entries from some tables, such as the security_hashes, sessions and cp_log as the information they contain is of no great use when moving from a development to a production server.

You can tell the script not to ignore those lines if you provide the —all_data or -a option.

Hopefully that clarifies the issue.

Wes Baker / 20th of March, 2009 / #13

AHA! Works perfectly now. Thanks for all your hard work.

Stephen / 20th of March, 2009 / #14

Glad to help. Thanks for your feedback.

Wes Baker / 20th of April, 2009 / #15

Having another problem with the script, it seems to be generating a huge regex that seems redundant. Here’s the command and the result.

Command:
root@ubuntuserver:/var/www/brp75# ruby /usr/bin/ee-migrate/ee_migrator.rb -a—current_domain=brp75—new_domain=brp75.newcitymedia.com—current_path=/Users/wes/Sites/brp75—new_path=/var/www/brp75 brp75.sql > brp75.new.sql


Result

Wes Baker / 20th of April, 2009 / #16

I realize the command I pasted doesn’t have a space after the -a but I tried it with a space there and got the same result.

Avi Block / 19th of May, 2009 / #17

I would like to add that I find it helpful to put as many configuration options outsite the database and into config.php. It becomes easier to edit them in a file, and works better with version control also. Any setting which is accessed using the $PREF->ini() method can be placed in config.php. This includes tmpl_file_basepath, show_queries/template_debugging, cookie_domain and others.

Yann / 12th of July, 2009 / #18

Brilliant… Thanks!

D. Shun-Luoi Fong / 22nd of August, 2009 / #19

I used this script once and it worked great for me. But recently I have tried to use the script, on Mac OS X terminal, and no changes are being made to the file. I use the following command:

ruby ee_migrator.rb—current_domain=currentdomain.com—new_domain=www.newdomain.com—current_path=/path/to/web—new_path=/new/path/to/web sql_dump.sql > live_sql.sql

After running the command I search the new file, live_sql.sql, for the new domain and it can’t be found. I search for the old domain and it is still there.

I’m sure I’m missing something very obvious, but can anyone suggest any reasons why this script wouldn’t be changing anything in the sql file?

Kevin / 26th of August, 2009 / #20

same problem here… the new file was created after I ran the script and a diff shows a few differences, but not the ones i was looking for :(

this is my cmd
~/ee_migrator.rb—current_domain=http://com.site.www—new_domain=http://dev.server.info—current_path=/Users/kevin/Sites/com.site.www—new_path=/home/kevin/dev.server.info/com.site.www ~/deploy/_sql/com_site_www.sql > ~/deploy/_sql/com_site_www_dev.sql

Stephen Rushe / 26th of August, 2009 / #21

For either of those I’d need to see the sql it’s being run on, rather than the command being used. Feel free to email it to me at whatto (at) deeden (dot) co (dot) uk and I’d be happy to have a look to see what the issue is.

chichilatte / 13th of October, 2009 / #22

A really neccessary utility, thanks! Sadly, it doesn’t seem to be able to migrate from a windows installation. If I do…

ruby ee_migrator.rb -c mysite.localhost -n http://mysite.webfaction.com -s c:/Projects/mysite -p /home/me/webapps/mysite local.sql > remote.sql


I get the error “The option—current_path must have a leading slash”. I’ll tinker with the ruby.

chichilatte / 13th of October, 2009 / #23

Ugg, it’s a whirlwind of jackson pollock inspired regexes! One day before I die I will settle in for a week and try to grasp that nettle. Perhaps a cottage by the sea in cornwall with a supply of whisky and bath salts.

Stephen / 26th of October, 2009 / #24

Chichilatte, if you try changing line 88 from

[ :current_path, :new_path ].each do |opt|

to

[ :new_path ].each do |opt|

it might do the needful. If so let me know and I’ll update the script to match.

Craig Vidler / 28th of February, 2010 / #25

Had tried the manual method you linked: fiddly, things missed, didn’t work. Your script: EE v1.6.8, Unix/MySQL 5.0.51 to OSX 10.5/MySQL 5.1.25, worked perfectly. Thank you! Craig