Sync a complete website to remote server quickly by Phing

The famous Phing build tool (written in PHP) offers automation for repetitive tasks, which need to be executed by the Linux terminal. So you can sync an entire website (files and database) to the remote server. Here is an example of a Phing target to do the job:

This target is later contained in your build.xml file. Afterwards you can execute the following command:

phing syncWebsiteToLive

The output will be the following:

Buildfile: /home/peter/build.xml
DevOps > syncWebsiteToLive:
     [echo] Syncing data to live server.
sending incremental file list
db_dumps/latest.sql
      7,515,285 100%  111.94MB/s    0:00:00 (xfr#1, ir-chk=1009/1040)
     [echo] Delete old database
     [echo] Create new empty database
     [echo] Import database dump
BUILD FINISHED
Total time: 5.7706 seconds

The website has been single by one single console command in an absolute minimum of time. This approach is good for a website in development stadium or a website which database can be just replaced. Then this way can save a lot of time compared to other approaches like ftp and manual database dump import.

Sync a complete website to remote server quickly by Phing

The famous Phing build tool (written in PHP) offers automation for repetitive tasks, which need to be executed by the Linux terminal. So you can sync an entire website (files and database) to the remote server. Here is an example of a Phing target to do the job:

This target is later contained in your build.xml file. Afterwards you can execute the following command:

phing syncWebsiteToLive

The output will be the following:

Buildfile: /home/peter/build.xml
DevOps > syncWebsiteToLive:
     [echo] Syncing data to live server.
sending incremental file list
db_dumps/latest.sql
      7,515,285 100%  111.94MB/s    0:00:00 (xfr#1, ir-chk=1009/1040)
     [echo] Delete old database
     [echo] Create new empty database
     [echo] Import database dump
BUILD FINISHED
Total time: 5.7706 seconds

The website has been single by one single console command in an absolute minimum of time. This approach is good for a website in development stadium or a website which database can be just replaced. Then this way can save a lot of time compared to other approaches like ftp and manual database dump import.

Sync a complete website to remote server quickly by Phing

The famous Phing build tool (written in PHP) offers automation for repetitive tasks, which need to be executed by the Linux terminal. So you can sync an entire website (files and database) to the remote server. Here is an example of a Phing target to do the job:

This target is later contained in your build.xml file. Afterwards you can execute the following command:

phing syncWebsiteToLive

The output will be the following:

Buildfile: /home/peter/build.xml
DevOps > syncWebsiteToLive:
     [echo] Syncing data to live server.
sending incremental file list
db_dumps/latest.sql
      7,515,285 100%  111.94MB/s    0:00:00 (xfr#1, ir-chk=1009/1040)
     [echo] Delete old database
     [echo] Create new empty database
     [echo] Import database dump
BUILD FINISHED
Total time: 5.7706 seconds

The website has been single by one single console command in an absolute minimum of time. This approach is good for a website in development stadium or a website which database can be just replaced. Then this way can save a lot of time compared to other approaches like ftp and manual database dump import.

Linux: See specific processes

ps aux | grep drush

Installing free of charge SSL certificate from Letsencrypt.org in Apache 2 webserver on Ubuntu 14.04

SSL certificates are cost-free nowadays. Letsencrypt.org is providing a service which lets you get SSL certificates that are free and work in the common web browsers. The following commands must be executed on the same machine on which your webserver is running. Because letsencrypt.org must be allowed to access it. Download the certbot from letsencrypt.org and make the binary executable:

wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto

Now temporary stop your Apache server, to download the certificates.

sudo service apache2 stop

Now run the certbot program to download the certificates:

./certbot-auto certonly

This program leads you trough a short form process within your terminal. Asking you for the domain you want to secure.

Mind the SSL certificate renewal

The SSL certificate is valid for 3 months. You will get notified about this via the terminal output from the certbot. It looks like this, if you have executed the "./certbot-auto certonly" command at 23.07.2016:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/my-domain.com/fullchain.pem. Your cert will
   expire on 2016-10-21. To obtain a new or tweaked version of this
   certificate in the future, simply run certbot-auto again. To
   non-interactively renew *all* of your certificates, run
   "certbot-auto renew"
 - If you like Certbot, please consider supporting our work by:
   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

There's an automatic way to accomplish the renewal: https://community.letsencrypt.org/t/how-to-automatically-renew-certificates/4393. Otherwise just make you a calendar entry and run the command from above again.

Enable the SSL certificate on your web server

Enable SSL in your Apache setup:

sudo a2enmod ssl

Now as you have the certificates, you can integrate them in your Apache 2 VHost configuration at /etc/apache2/sites-available/. Change "example.com" to your domain name.

   SSLEngine on
   SSLCertificateFile    /etc/letsencrypt/live/example.com/cert.pem
   SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
   SSLCertificateChainFile /etc/letsencrypt/live/example.com/fullchain.pem 

Switch the virtual host port from 80 to 443. For this set the following in your Vhost configuration file:
to

Small tweak for SEO

Search engines expect that a web page can be accessed by only one address on your website. To redirect all requests from http to https and ensure that only http://your-domain.com will be requested and not http://your-domain.com "and" http://www.your-domain.com, you can setup the redirect within your VHost file. Add this to the top of the configuration file for your virtual host:

Now restart Apache and happy SSL usage:

sudo service apache2 restart

MySQL: Import database dump

Firstly make sure that the "empty" database exists. Then import it:

mysql -uUSERNAME -p"PASSWORD" database_name < dump-filename.sql

Import a Gzip compressed database dump file:

zcat database-dump-file.sql.gz | mysql -uroot -ppassword database-name 

Git: Changing remote repository from one to the other

If you switch your Git repository from one to the other, you need to switch the address from the remote repository. To do so, you need the following command:

git remote set-url origin ssh://user@host:port/folder/repository-name.git

MySQL: dump database and compress it with Gzip

For accomplishing this, execute the following command:

mysqldump -u username -p"password" database_name | gzip > $(date +"%Y-%m-%d")-database_name.sql.gz

This command will generate you a compressed dump file with the date information in the filename.

Drupal 8: Execute automated tests and debug them in PhpStorm

To test Drupal 8 tests by the command line, you need to do the following. You need to export two environment variables first:

export SIMPLETEST_DB=mysql://root:password@localhost/database;
export SIMPLETEST_BASE_URL=http://host;

Then you can execute the test by the following command, being in the Drupal root folder:

vendor/bin/phpunit -c core modules/my_module/tests/src/Kernel/MyTest.php

Make sure to set the environment variable for PhpStorm:

export XDEBUG_CONFIG="remote_enable=1 remote_mode=req remote_port=9000 remote_host=127.0.0.1 remote_connect_back=0"

phpunit.xml 

There's a phpunit.xml.dist file in PHP projects. That's a template for your PHPUnit configuration. You can specifiy there some PHP ini settings. By this method you do not need to export the variables like above. Copy it to phpunit.xml and set there your settings like follows:

Afterwards you are able to run your test without the manual environment variables setting, just by specifying your configuration directory by the "-c" parameter. Example:

vendor/bin/phpunit -c modules/permissions_by_term modules/permissions_by_term/tests/src/Kernel/SelectTermTest.php

This command cannot execute SimpleTest tests, which inherit from the WebTestBase class. Only unit tests and tests which inherits from the KernelTestBase class can be run.

Runs a SimpleTest

// --- SimpleTests
php core/scripts/run-tests.sh --url http://d8-paragraphs:8888 --browser --verbose --color --concurrency 4 --class 'Drupal\paragraphs\Tests\ParagraphsPreviewTest'

Troubleshooting

If the debug mode is switched on in PhpStorm and the debugger does not stop at the breakpoint and you are getting the following output in the debugger window:

Remote file path '/home/user/Websites/my-project/-' is not mapped to any file path in project

If you have already setup path mapping, then there's probably any error in your code. Chances are, that it is in the first line of the setUp() method of your test class, which inherits from KernelTestBase. Then you probably do not have Drupal 8.2.x but Drupal 8.1.x. If it's in the first line, you can insert any code line or insert

xdebug_break(); 

With this function the debugger will be forced to stop on a specific line.

PhpStorm IDE settings (version 2016.1.2)

If you are executing the tests from the command line (and probably also via the IDE itself) it's important that you setup the path mappings, so that Xdebug can recognize your test's location. You can to this in PhpStorm's settings at "Languages & Frameworks > PHP > Servers". Create a host for your Drupal website host and set the path mapping to your tests directory. An example you can find in the screenshot below:

Linux: Get the size of a folder in an easily readable format

The du program on linux shows the sizes from folders nicely. Just type:

du -shc folder_name

 This will show you the folder size in an easy to read output (no bytes, but gigabytes):

peter@computer:/vol$ du -shc folder_name
3.5G    folder_name
3.5G    total

Easily re-format JSON files with the yajl-tools Ubuntu package

If you have a minimized JSON file which you want to reformat, then you can install the yajl-tools package on Ubuntu:

sudo apt-get install yajl-tools

Afterwards you are able to reformat the file with the following command:

json_reformat -s < ugly.json > pretty.json

PHP XPath and DOMDocument class: Encode URLs to valid HTML standard from a piece of HTML markup

The following code example shows how it works:

XDebug: Debug PHP scripts which are executed by Drush within your virtual machine

If you have PhpStorm (2016.1.1) installed on your host system and you want to debug PHP code in your virtual machine, which is executed by Drush (PHP command line application), so you need some configuration which is described below.

Settings in your PhpStorm IDE:

Map your project root folder to the absolute path on the server. Make sure that your webapplication's PHP code (e.g. Drupal) is located in the same folder as Drush. Like in the screenshot below.

Uncheck "Force break at the first line when a script is outside the project":

PhpStorm needs to listen to XDebug connections:

Afterwards you are able to Debug the code within Drush itself. But it is not necessarily needed to have the Drush folder mapped, when you want to only debug your Drupal code.

Install Drush via composer (PHP package manager)

composer require "drush/drush:7.3"

I have choosen a specific Drush version to suit in my webapplication's requirements. Run this command in the folder, which contains also your webapplication, to meet the path mappings:

/project-folder (run the command here)
  /webapplication
  /vendors

Drush is then located at "vendor/drush/drush". Make sure to run the Drush file which is located there by creating an alias in your .bash_profile file:

alias drush='php /var/www/vendor/drush/drush/drush.php'

Configure your PHP CLI (command line) settings, to work with XDebug

xdebug.remote_enable=true
xdebug.profiler_enable=0
xdebug.idekey=PHPSTORM
xdebug.max_nesting_level=256
xdebug.remote_autostart=true (THIS SETTING ENABLES/DISABLES XDEBUG)
xdebug.remote_connect_back=on
xdebug.remote_host=YOUR HOST IP. - E.G: 192.168.0.14

Force XDebug to break, so that you can set the path mapping

Use the following function in your code, to make XDebug connecting back to your IDE. If that works, you need only to set the path mapping within your PhpStorm IDE to be able to debug properly.

xdebug_break();

If everything works

Your IDE looks like below, if you have setup everything properly:

Now you can use the following Drush command to debug your code.

drush php-eval 'my_nice_function()'

Drupal will be booted up and you can inspect your code during the application runtime. This approach is faster then locating a certain page on your Drupal site for finding the entry point to your code or using the Devel module with it's webpage based PHP execution functionality at http://my-website.dev/devel/php.

Tip: Check out the Phing build tool which is written in PHP

I recommend to create a command with your favorite build tool (I recommend https://www.phing.info/) to quickly disable and enable XDebug. So that you execute "xd" for disabling and "xe" for enabling to save time.

Screen: Run processes in the background on a Linux server

The program "Screen" can run processes in background on a Linux server.

This starts an new background session with name "your-screen-name"

screen -S your-screen-name

To save the screen and return to the terminal, type Ctrl+A+D (Ctrl+A is the hint for screen that you want to do something and "D" then "d"etaches from the session without stopping it).

Restore your screen :

screen -d -r your-screen-name

List all available screen sessions:

screen -ls 

Type screen -list or -ls to identify the detached screen session.

~$ screen -list
    There are screens on:
         20751.Melvin_Peter_V42  (Detached)  

Note: 20751.Melvin_Peter_V42 is your session id.

Quit a screen session:

screen -X -S [session id you want to kill] quit

Or just type

exit

if you are attached in your screen session.

Linux Bash: Force writing console output to a file

Sometimes this command just does not write content to a file by the ">" operator:

myprogram do --something > my-file-with-the-output.txt

Then you can you a small Linux default program called "tee". Example usage:

myprogram do --something | tee my-file-with-the-output.txt

Tee allows you also to use some option as "-a" to extend the file's content's instead of replacing them.

Git: Delete a remote branch

By specifying "origin" and "--delete" as parameters according to the branch name (e.g. "feature/1234_my_ticket), you can delete the remote branch. The working Git command looks like this:

git push origin --delete my_name/1234_my_ticket_id

Apache2 RewriteRule: Map images folder from remote system to your local virtual host

With the following directive in your Apache2 virtual host file (at /etc/apache2/sites-available) you must not download the images from products etc. on your local system for development. You just map your local folder location to the remote one. The following example is for a Drupal site (files are under /sites/default/files):

RewriteEngine On
RewriteOptions Inherit
RewriteRule /sites/default/files/(.*)\.(jpeg|jpg|gif|png)$ http://www.my-website.com/sites/default/files/$1.$2 [L,R=301]

SFTP: Upload and download files from command line

Download one file:

sftp user@host:/path/to/file.txt localfile.txt

Upload one file:

scp localfile.zip user@host:/my/folder/

SSH: Quick way to add public key to ~/.ssh/authorized_keys file

cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

top: Filter linux/mac processes by name

The following command shows you, if e.g. dropbox is running:

top -b -n 1 | grep dropbox

Git: Setup on Windows file system

Git works also on Windows by the Git bash program, which integrates into other terminals, like the one from PhpStorm IDE. To use Git on Windows, you should be aware of a few issues, if you work with other file systems, like the ones from the Unix world (Linux/Mac).

Case insensitive file system

Windows has a case-insensitive filesystem. That means, that the filenames "facebook.jpg" and "Facebook.jpg" are the same file for Windows. If you have such files, you cannot regularly commit. You have to use a special command, to let Git assume this files are unchanged:

git update-index --assume-unchanged FILENAME

Afterwards you can commit and Git will not bother you with that file.

Unix file and folder permissions

Windows does not know about Unix file and folder permissions. Even you have not visibly changes inside your code, Git will mark all files as changed after clone from a Unix file system. To let Windows ignore the file mode use:

git config core.fileMode false

Line endings

Windows has a pair of CR and LF characters to terminate lines. Linux and Mac have a single LF character. To not have all files marked as changed after cloning on a Windows system, use the following setting:

git config --global core.autocrlf true

Creating a patch by Git

Firstly create a new Git branch: 

git checkout -b my-new-branch

Do your changes and commit:

git commit -am "My changes."

Now create the patch file by a diff from your branch and the development branch (f.e. master or develop):

git format-patch master --stdout > fix_empty_poster.patch 

Afterwards you can switch into a different branch, where you want to apply the patch file.

git checkout master

To check if the patch can be applied, use the following command:

git apply --check fix_empty_poster.patch 

To check which impact the patch will have, use the following command:

git apply --stat fix_empty_poster.patch

You can also apply the patch by

git apply < fix_empty_poster.patch

But this command will not create a Git commit by one action.

For applying the patch, use the following command:

git am --signoff < fix_empty_poster.patch 

 By this command, the patch will be already commited into Git. So you'll the the following entry in your Git log, if you type

git log

commit 2839f6f8e408140956d121b2af28b55543e93ec0
Author: Peter
Date:   Wed Mar 30 21:04:21 2016 +0200
    Patch commit.
 
    Signed-off-by: Peter

This article is based on Ariejan de Vroom's article.

The commands in short

New branch:

git checkout -b my-new-branch

Do changes and commit:

git commit -am "My changes."

Create the patch:

git format-patch master --stdout > fix_empty_poster.patch

Checkout to your master branch (or any other):

git checkout master

Check if patch can be applied:

git apply --check fix_empty_poster.patch 

Check the impact:

git apply --stat fix_empty_poster.patch

Apply and commit the patch:

git am --signoff < fix_empty_poster.patch 

Debugging PHP CLI scripts in PhpStorm which are running inside a Vagrant Box (Xdebug)

Please notice: This article is for debugging with Xdebug. Not any "old-fashioned" debugging with var_dump(); or print_r() or any other "90ies webmastering style".

Remote debugging by SSH tunnel and the PhpStorm debugger only

Additional settings in /etc/php5/cli/php.ini:

xdebug.remote_enable=true
xdebug.remote_autostart=true
xdebug.remote_port=9000
xdebug.remote_connect_back=1

Then connect to the server via SSH tunnel:

ssh -R 9000:localhost:9000 username_goes_here@hostname_goes_here 

Start a script via PHP from the command line and PhpStorm will react (debugging button in PhpStorm must be turned on and the mapping for the directories must be set).

This works, if you have not a weird issue with not available execution point inside PHPStorm's debugging window and there's no solution on the internet.. (Google isn't always your friend)

Remote debugging by the PhpStorm tools

In case that the debugger window will be opened, but the execution point will not be shown, you can run the script from PhpStorm itself.

Firstly you have to set up the remote debugging (Preferences > Languages & Framework > PHP). Choose here a remote interpreter.

If you have not already done so, set it up. The "SSH Credentials" settings have worked for me. While I'm using Vagrant and PhpStorm has an extra Vagrant checkbox (which has not worked for me):

In your PHP command line programs you need to specify often arguments. That can be done also:

sed: Another way of fixing wrong line endings on linux

sed is a stream editor which is pre-installed on linux systems like Ubuntu. It can fix line endings by the following command.

sed -i -e 's/\r$//' scriptname.sh

The manual can be found at http://www.gnu.org/software/sed/manual/sed.html. 

SSH: Fix folder and file permissions for ~/.ssh folder

SSH is checking the folder and files permissions at ~/.ssh. If there're any problems with reading or writing into that folder, the following command helps out.

chmod 700 ~/.ssh && chmod 600 ~/.ssh/*

Git: Fixing line-ending problem at Windows host system and Linux Guest (Vagrant)

To have the correct line endings in the Linux guest system by working on a Windows host, type the following Git command on the Windows host:

git config --global core.autocrlf true

git clean - Cleaning the Git repository to be able to merge with remote

Sometimes you get Git messages like

The following untracked working tree files would be overwritten by checkout

The following command can be helpful in that case.

git clean  -d  -fx ""

See http://www.kernel.org/pub/software/scm/git/docs/git-clean.html

-x means ignored files are also removed as well as files unknown to git.
-d means remove untracked directories in addition to untracked files.
-f is required to force it to run.

I've learned this by a discussion on Stackoverflow: http://goo.gl/I9NMxO.

rsync: Sync Two Folders via SSH in Both Directions

After started writing about rsync from local to live. Here's a post about syncing in two directions.

Helpful in rsync is, that also the permissions- and owner information is transfered ("-a" parameter). The "--progress" parameter shows the progress and "--exclude" allows to exclude directories. Be carefully about "--delete". Mind hidden folders like ".git" and ".vagrant" etc.

SSH session on local system (developer desktop) to sync files from (stage/live) server to local system

Here we assume you are in a ssh session on your local development environment and you want to sync from the stage/live server to your local development environment.

rsync -a source/ destination-host:/destination-directory/

Example:

rsync -a --progress www/ --exclude=sites/default/files/ --exclude=.idea/ --exclude=.vagrant/ --exclude=provisioning/ --exclude=.git ssh-username@host.com:/srv/www/destination-folder/

From (stage/live) server to local system (developer desktop)

Here we assume you are in a ssh session on your stage/live server and you want to sync from your local development environment to stage/live server.

Example:

rsync -chavzP --stats --progress username@host.com:/srv/www/some-folder/ /var/www/some-folder/

explainshell.com 

Detailed explanation for the commands can be found at explainshell.com. The rsync command above is explained here.

Happy syncing.

Drush: Rebuild the cache of a single Drupal (8) site in a multi-site setup with the magic "-l" parameter

After lots of googling, testing, reading and so on, I've finally found out the Drush command to flush the cache of a single Drupal (8) site in a multi-site setup.

drush -l 8888.notes cache-rebuild

If your site is located at /sites/8888.notes you need the command above. The "-l" parameter is necessary. I could not find out what "-l" means, but this parameter seemed necessary. 

GitHub not accepting public SSH key: fix it by switching from HTTP(S) to SSH in the Git repository config

If you clone a Git repository from GitHub via the HTTPS URL, which is placed in the section as shown below, than your public SSH key will not be recognized by GitHub.

You can test your public SSH key for GitHub usage by the following command:

ssh -T git@github.com

If you have chosen the wrong HTTPS option before (it must be SSH), you can fix that by the following steps.

1. Open the file at .git/config
2. Look for 
   url = http://github.com/path/to/repository or url = https://github.com/path/to/repository
3. Replace it by
   url = ssh://git@github.com/path/to/repository
4. Save the file and GitHub will accept the public SSH key on the next push. No username and password typing anymore.

Fix Line Endings Between Various File Systems

If you work sometimes on Windows, sometimes on Mac and sometimes on Linux, you can get troubles with line endings. Programs which aren't smart enough to detect line endings by them self, will display the text as if the file contained no line breaks at all.

• Windows, and DOS before uses a pair of CR and LF characters to terminate lines
• UNIX (Including Linux and FreeBSD) uses an LF character only. 
• OS X also uses a single LF character 
• Classic Mac operating system used a single CR character for line breaks. 

That's tricky. A tool for fixing line endings is "SFK - The Swiss File Knife File Tree Processor". It's mentioned at http://stackoverflow.com/questions/7068179/convert-line-endlings-for-whole-directory-tree-git/16886243#16886243.

All you have is to use the following command for converting to Mac OSX:

sfk remcr -dir your_project_directory

The project homepage is at http://stahlworks.com/dev/index.php?tool=remcrlf