How to Generate Expires Value for Amazon S3 or Google Cloud Storage Using Java

Google Cloud Storage and Amazon S3 supports signed URLs for protected objects. Signed URLs can be used to enable temporary public access to protected content. Anyone with the signed URL can access the protected resource. However the validity of the signed URL is specified by a query parameter named Expires. This parameter specifies the validity of the signed URL in terms of seconds elapsed since 1st January 1970 (also know as epoch).

Following is a sample signed URL to a protected Google cloud storage object (signature is truncated for brevity),

https://storage.googleapis.com/samplebucket/samplefile.jpg? GoogleAccessId=serviceuser@demoaccount-12345.iam.gserviceaccount.com& Expires=1487561198&Signature=nVmsboiEMOBZONx…

In the above URL, the Expires value specifies the validity of the URL as the number of seconds since 1st January 1970 (UTC).

Generating Expires Value in Java

The following simple method in Java can be used to generate Expires query parameter in an Amazon S3 or Google cloud storage signed URL. It will calculate an expiry time since the execution of the method itself. The parameter to the method is the number of seconds of validity of the signed URL since the execution of the method.

    // Return an Expires value calculated as number of seconds of validity from now!
    private long getExpiresForSecondsSinceNow(long seconds) {
        long now = System.currentTimeMillis();
        long expiryTime = (now + seconds*1000L)/1000; // convert millis to seconds
        return expiryTime;
    }

Note that the Expires field in the signed URL query parameter must be the same one specified in the original signed text used for generation of signed URL. If you are using signed URLs for delivering protected media content such as video or images, it is important to keep a short expiry time such as 60 seconds.

Buildship 2.0 and Spring STS 3.8.3

Spring STS 3.8.3 doesn’t work well with Buildship 2.0 Gradle plugin. When you try to create a new Spring starter project with Buildship Gradle plugin selected, you will get the following error,

can not import using Gradle (Buildship) because Buildship Gradle Tooling is not installed. You can install it from Eclipse Marketplace.

Note that you will get this error even after installing Buildship 2.0 from the Eclipse marketplace. This is actually an STS bug since it had a Buildship version dependency built into it. Luckily this bug is fixed in the upcoming STS 3.8.4 release.

There are two solutions to this problem. You can either get a nightly build of the upcoming STS 3.8.4 release or you can install an older version of the Buildship plug-in.

Option 1: Installing Nightly Build of STS 3.8.4

The quickest solution to the above bug is to install the latest nightly build of the upcoming STS 3.8.4 release. The only downside of this approach is that 3.8.4 itself may have other bugs! To install the nightly build in Spring STS, click on Help => Install New Software. In the work with field, add the URL to the Spring STS nightly build for Eclipse Neon. Install the updates listed.

Option 2: Installing Older Version of Buildship

If you have Buildship 2.0 already installed, uninstall it from Help => Installation Details. Then click on Help => Install New Software. In the work with field, add the URL to the Buildship 1.0.21 plug-in for Eclipse Neon. Install the Buildship 1.0.21 version listed. Voila! Now you can create Spring starter projects using Buildship.

References

Bug report in GitHub

Automatic Backup of Compute Engine Persistent Disks in Google Cloud Platform

There are two ways to create a backup snapshot of compute engine persistent disks. You can either take a snapshot from the Google cloud console or run gcloud snapshot command from the command line.  The following command creates a snapshot (backup) of the disk named "mydisk1" created in "asia-east1-a" availability zone,

gcloud compute disks snapshot mydisk1 –zone=asia-east1-a

If you want to enable automatic daily or weekly backups, you need to add the above command in a cron job. The cron job for automatic backups can be run in a VM in the compute engine or in a Docker container in Google Container Engine. Configuring the above command in cron requires the following steps,

  1. Install Google Cloud SDK (required for the gcloud command). If you are configuring cron in a Linux VM, follow these instructions to install Google Cloud SDK. If you are using Google Container Engine, you can create a custom container with Google Cloud SDK from this base image. Such a container can be used for all maintenance activities of your production environment.
  2. Create a file named cronfile with the following content,

    PATH=/google-cloud-sdk/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin
    0 23 * * * /myfolder/backup.sh

    The above cronfile will run the backup.sh every day at 11pm. Note the use of PATH variable. Please change the first path element to point to the folder where gcloud command is located.

  3. Create the backup.sh script with the following content (don’t forget to change zone for your disk),

    gcloud compute disks snapshot mydisk1 –zone=asia-east1-a

    Ensure that the backup.sh is executable by running the following command,

    chmod +x backup.sh
  4. Finally enable the cronfile by configuring it as a cron job for a user account.

    crontab -u root cronfile
    cron reload

How to Copy Files from One Folder to Another in Java

The following Java program copies files with .java extension from one folder to another folder. You can easily change the extension .java to anything you want. For example, you might want to copy just .txt files from one directory to another.

This program takes two input parameters. The first one is the source directory and the second one is the target directory. Please ensure that you enter full path of the directories. This program also displays errors encountered if any during the file copying.

Note that this program doesn’t delete the source file. This program only uses the File and File stream classes.

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FilenameFilter;
import java.io.InputStream;
import java.io.OutputStream;

/**
* Copy all the files in a folder with specific extension to another folder. 
* Both the folders are specified on the command line. 
* The program currently uses .java extension as filter.
**/
public class CopyFilesWithExtension {
	
	public static void main(String[] args) {
		if(args.length!=2) {
			System.out.println("Command usage: java CopyFilesWithExtension sourcedirectory targetdirectory");
		}
		
		String sourceFolder = args[0];
		String targetFolder = args[1];
		
		File sFile = new File(sourceFolder);
		// Find files with specified extension
		File[] sourceFiles = sFile.listFiles(new FilenameFilter() {
			
			@Override
			public boolean accept(File dir, String name) {
				if(name.endsWith(".java")) {// change this to your extension
					return true;
				}else {
					return false;
				}
			}
		});
		
		// let us copy each file to the target folder
		for(File fSource:sourceFiles) {
			File fTarget = new File(new File(targetFolder), fSource.getName());
			copyFileUsingStream(fSource,fTarget);
			// fSource.delete(); // Uncomment this line if you want source file deleted
		}
		
		
	}
	
	/**
	 * Copies a file using the File streams
	 * @param source
	 * @param dest
	 */
	private static void copyFileUsingStream(File source, File dest)  {
	    InputStream is = null;
	    OutputStream os = null;
	    try {
	        is = new FileInputStream(source);
	        os = new FileOutputStream(dest);
	        byte[] buffer = new byte[1024];
	        int length;
	        while ((length = is.read(buffer)) > 0) {
	            os.write(buffer, 0, length);
	        }
	    }catch(Exception ex) {
	    	System.out.println("Unable to copy file:"+ex.getMessage());
	    }	
	    finally {
	    	try {
	    		is.close();
	    		os.close();
	    	}catch(Exception ex) {}
	    }
	}

}

Array to Map Conversion in Java

Converting an array of strings to a map of strings with the same string as key and value is required in many use cases. Conversion to map enables quick and easy evaluation as to whether a string exists in the list of strings. Conversion to map also enables removal of duplicate strings. The following Java program demonstrates conversion of an array of strings to its corresponding map.

import java.util.HashMap;
import java.util.Map;

/**
 * Converts an array of strings to a map. Each string becomes the key and the corresponding value.
 * Useful for removal of duplicates and quick check on existence of a string in the list.
 * @author jj
 */
public class ArrayToMap {
    
    public static void main(String[] args) {
        String[] colors = new String[]{"blue","green","red"};
        Map<String,String> colorMap = new HashMap<String,String>();
        for(String color:colors) {
            colorMap.put(color, color);
        }
    }
    
}