Add Travis-CI config for online Linux Build server:

Merges all the commits from the travis-config branch into master. Summary:
New python script to pack the project folder into a zip file located into the upload folder.
Update the build_linux.py script to be more verbose and have better error detection.
Updated package README with builds information.

Squashed commit list:

commit 0190859b28f54c2fc73905125ec148329a2f1323
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Thu Jun 4 01:33:00 2015 +0100

    Removed the travis configure package bash script as no longer used.

commit 974e2ff05712a159c5e6a22779f589100173f1c6
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Thu Jun 4 01:29:51 2015 +0100

    Travis: Changed S3 bucket name, updated package readme.

commit 166ab734ae5d65161edfd641be3a9473ededec03
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 20:04:14 2015 +0100

    Travis: moved env var checks to pack_ardublockly script.

commit a9a752c49623f7462c1b80b0648ff6d9ded3cc2d
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 18:53:13 2015 +0100

    Travis: edit yml file to include pack bash code

commit 3821bb6b86e8cdcd54616a0816200035f3eddc43
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 18:20:59 2015 +0100

    Travis: edited the config yml file with right chmod command

commit 94c2daea15ef3b96c89bf232283779fc484d13d4
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 17:48:55 2015 +0100

    Travis: Add execution rights to pack bash script

commit b47462f1aef95361b27ee2b880d2a3c0b1362527
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 16:06:05 2015 +0100

    Travis: configure pack bash script needs sudo

commit 7cdcac4b8e7c6aea25598945cc54cd7deaaa03de
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 09:47:31 2015 +0100

    Travis: configure packing tag dependant on travis request

commit a55e41a23898e55536dac4747dd93380e6e5c412
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 09:30:26 2015 +0100

    Travis: Add tags to the zip build uploaded to S3

commit 6845419b5a728af25c232280f8b43c88aaf41e2f
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 08:24:19 2015 +0100

    Travis: add the command to skip cleanup before deployment

commit 20732f74494e52c2b2a51068c36bfbd395616bff
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 08:10:55 2015 +0100

    Server build scripts updates:
    Update the build_linux.py script to be more verbose.
    Add a script to pack the ardublockly folder into a zip file
    Update travis ci config file to upload to S3 only the zipped file.
    Update gitignore to ignore the zip file new folder names "upload".

commit 0194dcbccc5ae793a8ae1a7a7bf1262c088b81de
Merge: 22be32f b044094
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 04:58:32 2015 +0100

    Merge branch 'travis-config' of https://github.com/carlosperate/ardublockly into travis-config

commit 22be32f355e8dd4bcaa3849f098d0826e1f17081
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 04:57:49 2015 +0100

    Update PyInstaller build script to print more info.

commit b044094d47de07db4c350b555f935a1183be5efe
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 00:54:14 2015 +0100

    Travis: rebuild before deployment

commit fadd45179064f7b8721ae5e50fd8f1d7859ef18f
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Wed Jun 3 00:02:53 2015 +0100

    Travis: test needs to be run in script key

commit d41054d44dc47fe527d587f25237f638e6066bb9
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Tue Jun 2 23:48:13 2015 +0100

    Travis: edit test slashes for linux

commit 686bfb4eb67b8eff9d9b385b594344a97decfaa0
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Tue Jun 2 23:40:49 2015 +0100

    Adding test execution to travis-ci

commit 5ce69235a270358ec15b6f4706b7b67da2b0da49
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Tue Jun 2 23:27:31 2015 +0100

    Adding deployment options to travis-ci

commit 4e32944f5f7ea4cadbe8c7e2e41a29fdf9e5ab31
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Tue Jun 2 20:33:58 2015 +0100

    travis-ci: reduce the python path injection

commit 1a6fdcdf8911376d13c0391b0c814549dfef8334
Author: carlosperate <carlosperate@users.noreply.github.com>
Date:   Tue Jun 2 20:17:41 2015 +0100

    Adding extra python paths for travils-ci

.gitignore

@@ -85,6 +85,7 @@ cef_debug.log
 /ignore
 
 # Packaged python build folders and files
+/upload
 /win
 /arduexec
 /package/build

.travis.yml

 language: python
 python:
-  - "2.7"
-# Install wxPython and cefpython3
+- '2.7'
+
+# Install wxPython and cefpython3 (cef from official dropbox link)
 before_install:
-  - sudo apt-get update -qq
-  - sudo apt-get install python-support
-  - sudo apt-get install -y python-wxgtk2.8 python-wxtools wx2.8-i18n
-  - sudo wget https://www.dropbox.com/sh/zar95p27yznuiv1/AACGmGx08UMq8uEGDFlINFdwa/31.2/Linux/python-cefpython3_31.2-1_amd64.deb?dl=1
-  - sudo dpkg -i python-cefpython3_31.2-1_amd64.deb?dl=1
-# command to run tests, e.g. python setup.py test
-install:  python package/build_linux.py
+- sudo apt-get update -qq
+- sudo apt-get install python-support
+- sudo apt-get install -y python-wxgtk2.8 python-wxtools wx2.8-i18n
+- sudo wget https://www.dropbox.com/sh/zar95p27yznuiv1/AACGmGx08UMq8uEGDFlINFdwa/31.2/Linux/python-cefpython3_31.2-1_amd64.deb?dl=1
+- sudo dpkg -i python-cefpython3_31.2-1_amd64.deb?dl=1
+
+# Run the build script
+# Adding cefpython installation dir to travis-ci python paths, listed in:
+# https://code.google.com/p/cefpython/wiki/Download_CEF3_Linux
+install:
+- export PYTHONPATH=$PYTHONPATH:/usr/share/pyshared/
+- export PYTHONPATH=$PYTHONPATH:/usr/lib/pymodules/python2.7/
+- python package/build_linux.py
+- python package/pack_ardublockly.py
+
+# For now only running a known passing test, the others need to be updated
+script:
+- python ArdublocklyServer/tests/SketchCreator_test.py 
+
+# Deploy the build version in an S3 bucket
+deploy:
+  provider: s3
+  access_key_id: AKIAIBDOPX6GSMBDEFPQ
+  secret_access_key:
+    secure: QNNNIgHpcTYJZS1K4w8iPqJWq1Xhx8ICoVJ79QURS5Vi4ppnP0smuW13+XdmjoZyoceSoR4iOcKX7o4HXDozsejy3ft72tXgNntVQoOOKYxnV1zKYHc7Q6ID6L3GlPKV2EQid30CM4bP0mm4P8ntuemUOwhqm4mjRwDxAMpxsEbXgNAHAMifn40g5u2rHuwo64+UUll6m2Ues6mekAlwq6YQKn0numrcHRZX9F3kMLyrSHO671EkyfVsBbZpk7+HIEXmgXgPb4juefcxUlhYQFvnlBPyyTG1ec0BE2un714vZe+Gq0w1UYQbIpsAvM1HDWQKa2S9m2gz1TXC5YXOKAfoenyFsrgvdT00zpxVrTLAGhmIgtxsbAYXVPRg/6g5E+DW/JJ2xBfMJyw3j/82vexkHpvQcut9zZ74CE2H0CtyNXHvwdcwMs+qoja6tLw1dvFD7yrTo2T/jCOxFoGYjneF3Syuh8GpALSGGUJDnyFiK0yD2B0r2567csFj5Sc0CO/ExFwfG9zmiVtkm0Xpfq09VcZFjVnJ4k8QyNvrHmnGB/H7P6CskSA1X9gGIyk36Cu17XyyrtIuM+UUIdnrZhWCI3zA/2UVawkEjA+zhpbKFKsxIe37bPpIpNM+OyMLppZzzpcaRAynh6a/8Hcc+ckh4zMg7qWu7VXyYvjSkl0=
+  bucket: ardublockly-builds
+  skip_cleanup: true
+  local-dir: upload
+  upload-dir: linux
+  acl: public_read
+  on:
+    repo: carlosperate/ardublockly

package/README.md

@@ -3,7 +3,13 @@ This folder contains the Python scripts required to package Ardublockly into a s
 
 Currently the Windows build is created using py2exe and the Linux and Mac OS X builds are created using PyInstaller (in the future PyInstaller might be updated to also create the Windows builds, for now the original py2exe script is pretty stable). 
 
-__Downloading the packaged Ardublockly__: The binaries will be hosted in GitHub as part of the [repository releases][1].
+
+## Download the packaged Ardublockly
+The stable binaries for Windows, Linux an Mac OS X are hosted in GitHub as part of the [repository releases][1].
+
+Development builds are frequently triggered in a build server and hosted in the following link: [Dev builds][10]
+
+Linux build status: [![Build Status](https://travis-ci.org/carlosperate/ardublockly.svg?branch=master)](https://travis-ci.org/carlosperate/ardublockly)
 
 
 ## Building Ardublockly from source
@@ -115,3 +121,4 @@ This part of the documentation is still under work.
 [7]: https://github.com/carlosperate/ardublockly/wiki
 [8]: http://www.embeddedlog.com/static-docs-from-github-wiki.html
 [9]: https://github.com/pyinstaller/pyinstaller/wiki
+[10]: http://ardublockly-builds.s3-website-us-west-2.amazonaws.com/index.html
\ No newline at end of file

package/build_linux.py

@@ -17,6 +17,15 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+# This script file uses PyInstaller to create a self contained executable
+# build of the Ardublockly application.
+# It will remove the build folders left from PyInstaller and move the folder
+# with the executable application into the project root folder.
+#
+# Due to all the debugging required to get a CI serve running properly this
+# script ended up being quite verbose. In might be updated in the future to
+# include a -v flag to select a verbose mode.
+from __future__ import unicode_literals, print_function
 import os
 import shutil
 import subprocess
@@ -31,14 +40,24 @@ script_tag = "[Ardublockly build] "
 script_tab = "                    "
 
 
+#verboseprint = print if verbose else lambda *a, **k: None
+
+
 def remove_pyinstaller_temps():
     """ Removes the temporary folders created by PyInstaller. """
     dist_dir = os.path.join(os.getcwd(), "dist")
-    build_dir = os.path.join(os.getcwd(), "build")
     if os.path.exists(dist_dir):
+        print(script_tab + "Removing %s folder." % dist_dir)
         shutil.rmtree(dist_dir)
+    else:
+        print(script_tab + "No dist %s found." % dist_dir)
+
+    build_dir = os.path.join(os.getcwd(), "build")
     if os.path.exists(build_dir):
+        print(script_tab + "Removing %s folder." % build_dir)
         shutil.rmtree(build_dir)
+    else:
+        print(script_tab + "No %s folder found." % build_dir)
 
 
 def pyinstaller_build():
@@ -69,14 +88,27 @@ def remove_executable_folder():
     """ Removes the current ardublockly PyInstaller executable folder. """
     exec_dir = os.path.join(project_root_dir, exec_folder_name)
     if os.path.exists(exec_dir):
+        print(script_tab + "Removing the %s folder." % exec_folder_name)
         shutil.rmtree(exec_dir)
+    else:
+        print(script_tab + "No %s folder found." % exec_folder_name)
 
 
 def move_executable_folder():
-    """ Moves the PyInstaller executable folder from dist to project root. """
+    """
+    Moves the PyInstaller executable folder from dist to project root.
+    :return: Boolean indicating the success state of the operation.
+    """
     original_exec_dir = os.path.join(project_root_dir, "dist", exec_folder_name)
     if os.path.exists(original_exec_dir):
+        print(script_tab + "Moving folder %s \n" % original_exec_dir +
+              script_tab + "to %s" % project_root_dir)
         shutil.move(original_exec_dir, project_root_dir)
+    else:
+        print(script_tab + "ERROR: PyInstaller executable output folder not "
+                           "found!")
+        return False
+    return True
 
 
 def copy_cefpython_data_files():
@@ -100,6 +132,8 @@ def copy_cefpython_data_files():
         os.makedirs(cef_exec_locales)
 
     # Copy all the files
+    print(script_tab + "Copying CEF files from %s\n" % cef_path +
+          script_tab + "into %s" % cef_exec_folder)
     for f in data_files:
         shutil.copy(f, cef_exec_folder)
     for f in locales:
@@ -115,9 +149,10 @@ def create_bash_file():
                 'DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )\n' \
                 'echo "[Bash File] Running Ardublockly from: " $DIR\n' \
                 './' + exec_folder_name + '/ardublockly $DIR\n'
+    bash_location = os.path.join(project_root_dir, "ardublockly_run.sh")
     try:
-        bash_file = open(
-            os.path.join(project_root_dir, "ardublockly_run.sh"), 'w')
+        print(script_tab + "Creating bash file into %s" % bash_location)
+        bash_file = open(bash_location, 'w')
         bash_file.write(bash_text)
         bash_file.close()
     except Exception as e:
@@ -126,7 +161,7 @@ def create_bash_file():
                            "application could not be created.")
 
 
-def main():
+def build_ardublockly():
     print(script_tag + "Building Ardublockly for Linux.")
     print(script_tag + "Project directory is:     %s" % project_root_dir)
     print(script_tag + "Script working directory: %s" % os.getcwd())
@@ -137,16 +172,21 @@ def main():
     print(script_tag + "Running PyInstaller process:")
     success = pyinstaller_build()
     if not success:
-        print(script_tag + "Removing PyInstaller recent temp directories.")
+        print(script_tab + "Removing PyInstaller recent temp directories.")
         remove_pyinstaller_temps()
-        raise SystemExit(script_tag + "Exiting now as there was an error in "
-                                      "the PyInstaller execution.")
+        raise SystemExit(script_tab + "Exiting as there was an error in the "
+                                      "PyInstaller execution.")
 
     print(script_tag + "Removing old ardublockly executable directory.")
     remove_executable_folder()
 
     print(script_tag + "Moving executable folder to project root.")
-    move_executable_folder()
+    success = move_executable_folder()
+    if not success:
+        print(script_tab + "Removing PyInstaller recent temp directories.")
+        remove_pyinstaller_temps()
+        raise SystemExit(script_tab + "Exiting now as there was an error in "
+                                      "the PyInstaller execution.")
 
     print(script_tag + "Coping cefpython data files into executable directory.")
     copy_cefpython_data_files()
@@ -159,4 +199,4 @@ def main():
 
 
 if __name__ == "__main__":
-    main()
+    build_ardublockly()

package/pack_ardublockly.py

+#!/usr/bin/env python2
+# -*- coding: utf-8 -*- #
+#
+# Creates a zip file of the self executable Ardublockly application.
+#
+# Copyright (c) 2015 carlosperate https://github.com/carlosperate/
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This script file will create a copy of the project folder in its parent folder
+# So if project folder is located in ~/project/ardublockly it will create a
+# copy in ~/project/ardublockly_zip .
+# It will then delete unnecessary files for a working version of the self
+# executable application and zip the contents of the folder.
+#
+import os
+import sys
+import time
+import shutil
+import zipfile
+
+# The project_root_dir depends on the location of this file, so it cannot be
+# moved without updating this line
+project_root_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
+copy_dir_name = "ardublockly_packed"
+copied_project_dir = os.path.join(os.path.dirname(project_root_dir),
+                                  copy_dir_name)
+script_tag = "[Ardublockly pack] "
+script_tab = "                   "
+
+
+def set_tag(tag):
+    """
+    Sets the packaged zip file and copied folder tag to the input argument. So,
+    the copied folder will be names "ardublockly_<tag>" and the zip file
+    "ardublockly_<tag>.zip.
+    :tag: String to indicate the tag to use.
+    """
+    print(script_tag + "Setting the Ardublockly package tag to '%s'" % tag)
+    global copy_dir_name
+    global copied_project_dir
+    copy_dir_name = "ardublockly_%s" % tag
+    copied_project_dir = os.path.join(os.path.dirname(project_root_dir),
+                                      copy_dir_name)
+
+
+def copy_ardublockly_folder():
+    """
+    Copies all the contents of the project root directory into a new folder on
+    the same level.
+    The copy operation ignores a list of directories.
+    :return: Boolean indicating the success state of the operation.
+    """
+    ignore_pat = ("*.pyc", ".git*", "TestTemp_*", ".svn", "package", ".travis*",
+                  "python-cefpython3*")
+
+    if not os.path.exists(copied_project_dir):
+        print(script_tab + "Copying contents of %s\n" % project_root_dir +
+              script_tab + "into                %s" % copied_project_dir)
+        shutil.copytree(project_root_dir,
+                        copied_project_dir,
+                        symlinks=False,
+                        ignore=shutil.ignore_patterns(*ignore_pat))
+    else:
+        print(script_tab + "ERROR: %s directory already exists!" %
+              copied_project_dir)
+        return False
+    return True
+
+
+def remove_directory(dir_to_remove):
+    """ Removes the a given directory. """
+    if os.path.exists(dir_to_remove):
+        print(script_tab + "Removing directory %s" % dir_to_remove)
+        shutil.rmtree(dir_to_remove)
+    else:
+        print(script_tab + "Directory %s was not found." % dir_to_remove)
+
+
+def zip_ardublockly_copy(name_append):
+    """
+    Zips the contents of the copied project folder into a subdirectory of
+    the original project folder.
+    """
+    zip_file_dir = os.path.join(project_root_dir, "upload")
+    zip_file_location = os.path.join(
+        zip_file_dir, "ardublockly_%s.zip" % name_append)
+
+    # First ensure the upload folder exists
+    if not os.path.exists(zip_file_dir):
+        os.makedirs(zip_file_dir)
+
+    os.chdir(os.path.dirname(project_root_dir))
+    print(script_tab + "Working directory changed to %s" % os.getcwd())
+    print(script_tab + "Zipping the contents of %s\n" % copied_project_dir +
+          script_tab + "into                    %s\n" % zip_file_location)
+
+    zip_file = zipfile.ZipFile(zip_file_location, "w", zipfile.ZIP_DEFLATED)
+    for dir_name, sub_dirs, files in os.walk(copy_dir_name):
+        zip_file.write(dir_name)
+        for filename in files:
+            zip_file.write(os.path.join(dir_name, filename))
+    zip_file.close()
+
+
+def pack_ardublockly(tag):
+    # Set the copied folder name to the stamp
+    set_tag(tag)
+
+    print(script_tag + "Copying the project root folder:")
+    success = copy_ardublockly_folder()
+    if not success:
+        raise SystemExit(script_tab + "Exiting due to project root copy error.")
+
+    print(script_tag + "Removing unnecessary Blockly files:")
+    remove_directory(os.path.join(copied_project_dir, "blockly", "demos"))
+    remove_directory(os.path.join(copied_project_dir, "blockly", "appengine"))
+    remove_directory(os.path.join(copied_project_dir, "blockly", "tests"))
+
+    print(script_tag + "Removing an already zipped Ardublockly version:")
+    remove_directory(os.path.join(copied_project_dir, "upload"))
+
+    print(script_tag + "Removing CEF temporary files:")
+    remove_directory(os.path.join(copied_project_dir, "webcache"))
+
+    print(script_tag + "Creating zip file of the new Ardublockly folder:")
+    zip_ardublockly_copy(tag)
+
+
+def tag_from_travis_env_vars():
+    """
+    Checks if the Travis-CI environmental variables to check for a pull request,
+    commit id and band commit branch are present.
+    :return: String with the Travis build information, or None if the Travis-CI
+             environmental variables could not be found.
+    """
+    travis_pull_request = os.environ.get("TRAVIS_PULL_REQUEST")
+    travis_branch = os.environ.get("TRAVIS_BRANCH")
+    travis_commit = os.environ.get("TRAVIS_COMMIT")
+
+    if travis_pull_request:
+        if travis_pull_request != "false":
+            return travis_pull_request
+        elif travis_branch and travis_commit:
+            return "%s_%s" % (travis_branch, travis_commit)
+        else:
+            print(script_tab + "TRAVIS_BRANCH or TRAVIS_COMMIT Travis-CI"
+                               "environmental variables not found.")
+            return None
+    else:
+        print(script_tab + "TRAVIS_PULL_REQUEST Travis-CI environmental "
+                           "variable not found.")
+        return None
+
+
+def main():
+    print(script_tag + "Pack Ardublockly script started.")
+    print(script_tag + "Checking command line arguments for tag:")
+    # Check if a command line argument has been given
+    if len(sys.argv) > 1:
+        # Take the first argument and use it as a filename tag
+        tag = sys.argv[1]
+        print(script_tab + "Command line argument '%s' found and will be used "
+                           "for package tag." % sys.argv[1])
+    else:
+        print(script_tab + "No command line argument found")
+        print(script_tag + "Checking Travis-CI environment variables for tag:")
+        tag = tag_from_travis_env_vars()
+        if not tag:
+            # If the Travis-CI environmental variables are not present then
+            # use the current time stamp
+            tag = time.strftime("%Y-%m-%d_%H.%M.%S")
+            print(script_tag + "No Travis-CI environment variables found, so "
+                               "using current timestamp for tag: %s" % tag)
+
+    pack_ardublockly(tag)
+
+
+if __name__ == "__main__":
+    main()