I found a really cool github project which extends the bash utility awk to handle JSON objects in bash. Enter Jsawk
Installation
Installing the utility is pretty easy, we pretty much grab it from github
cd ~curl -L http://github.com/micha/jsawk/raw/master/jsawk > jsawk
Next we make it executable and put it somewhere we can access it
chmod 755 jsawk && mv jsawk ~/bin/
Dependencies
One dependency which jsawk has if you are running this on Mac is a JS binary. OSX doesn’t seem to have one already installed so lets go and grab one. Enter SpiderMonkey.
SpiderMonkey is a JavaScript engine used in FireFox, so lets download it and put it somewhere sensible
cd ~/binwget http://web.stanford.edu/class/cs242/spidermonkey/js-osx.zipunzip js-osx.zip
That should give you an executable utility called js, now we can tell jsawk about it with the following switch
jsawk -j <path to spidermonkey js>
Alias
To make things easy lets add an alias so that we dont need to keep adding this switch when we want to use the utility
vim ~/.bash_profile #addalias jsawk='~/bin/jsawk -j ~/bin/js'
Now just running jsawk will work without any problems
Usage
Here are some examples of usage.
Below is an example JSON document from an output of AWS CLI command to create 2 EC2 instances in AWS
run-instances.json
{
"OwnerId": "999412538050",
"ReservationId": "r-3332b3d7",
"Groups": [],
"Instances": [
{
"Monitoring": {
"State": "disabled"
},
"PublicDnsName": null,
"KernelId": "aki-52a34525",
"State": {
"Code": 0,
"Name": "pending"
},
"EbsOptimized": false,
"LaunchTime": "2015-05-28T08:34:30.000Z",
"PrivateIpAddress": "172.31.25.13",
"ProductCodes": [],
"VpcId": "vpc-6bc9110e",
"StateTransitionReason": null,
"InstanceId": "i-ca18d560",
"ImageId": "ami-1d9eee6a",
"PrivateDnsName": "ip-172-31-25-13.eu-west-1.compute.internal",
"KeyName": "orc-server",
"SecurityGroups": [
{
"GroupName": "Orc Server",
"GroupId": "sg-2903204c"
}
],
"ClientToken": null,
"SubnetId": "subnet-8024b3e5",
"InstanceType": "m3.medium",
"NetworkInterfaces": [
{
"Status": "in-use",
"MacAddress": "02:ca:99:88:4d:97",
"SourceDestCheck": true,
"VpcId": "vpc-6bc9110e",
"Description": null,
"NetworkInterfaceId": "eni-6850770c",
"PrivateIpAddresses": [
{
"PrivateDnsName": "ip-172-31-25-13.eu-west-1.compute.internal",
"Primary": true,
"PrivateIpAddress": "172.31.25.13"
}
],
"PrivateDnsName": "ip-172-31-25-13.eu-west-1.compute.internal",
"Attachment": {
"Status": "attaching",
"DeviceIndex": 0,
"DeleteOnTermination": true,
"AttachmentId": "eni-attach-6164ae20",
"AttachTime": "2015-05-28T08:34:30.000Z"
},
"Groups": [
{
"GroupName": "Orc Server",
"GroupId": "sg-2903204c"
}
],
"SubnetId": "subnet-8024b3e5",
"OwnerId": "999412538050",
"PrivateIpAddress": "172.31.25.13"
}
],
"SourceDestCheck": true,
"Placement": {
"Tenancy": "default",
"GroupName": null,
"AvailabilityZone": "eu-west-1b"
},
"Hypervisor": "xen",
"BlockDeviceMappings": [],
"Architecture": "x86_64",
"StateReason": {
"Message": "pending",
"Code": "pending"
},
"RootDeviceName": "/dev/sda1",
"VirtualizationType": "paravirtual",
"RootDeviceType": "ebs",
"AmiLaunchIndex": 1
},
{
"Monitoring": {
"State": "disabled"
},
"PublicDnsName": null,
"KernelId": "aki-52a34525",
"State": {
"Code": 0,
"Name": "pending"
},
"EbsOptimized": false,
"LaunchTime": "2015-05-28T08:34:30.000Z",
"PrivateIpAddress": "172.31.25.12",
"ProductCodes": [],
"VpcId": "vpc-6bc9110e",
"StateTransitionReason": null,
"InstanceId": "i-cb18d561",
"ImageId": "ami-1d9eee6a",
"PrivateDnsName": "ip-172-31-25-12.eu-west-1.compute.internal",
"KeyName": "orc-server",
"SecurityGroups": [
{
"GroupName": "Orc Server",
"GroupId": "sg-2903204c"
}
],
"ClientToken": null,
"SubnetId": "subnet-8024b3e5",
"InstanceType": "m3.medium",
"NetworkInterfaces": [
{
"Status": "in-use",
"MacAddress": "02:ba:95:c0:5d:a3",
"SourceDestCheck": true,
"VpcId": "vpc-6bc9110e",
"Description": null,
"NetworkInterfaceId": "eni-6950770d",
"PrivateIpAddresses": [
{
"PrivateDnsName": "ip-172-31-25-12.eu-west-1.compute.internal",
"Primary": true,
"PrivateIpAddress": "172.31.25.12"
}
],
"PrivateDnsName": "ip-172-31-25-12.eu-west-1.compute.internal",
"Attachment": {
"Status": "attaching",
"DeviceIndex": 0,
"DeleteOnTermination": true,
"AttachmentId": "eni-attach-4164ae00",
"AttachTime": "2015-05-28T08:34:30.000Z"
},
"Groups": [
{
"GroupName": "Orc Server",
"GroupId": "sg-2903204c"
}
],
"SubnetId": "subnet-8024b3e5",
"OwnerId": "999412538050",
"PrivateIpAddress": "172.31.25.12"
}
],
"SourceDestCheck": true,
"Placement": {
"Tenancy": "default",
"GroupName": null,
"AvailabilityZone": "eu-west-1b"
},
"Hypervisor": "xen",
"BlockDeviceMappings": [],
"Architecture": "x86_64",
"StateReason": {
"Message": "pending",
"Code": "pending"
},
"RootDeviceName": "/dev/sda1",
"VirtualizationType": "paravirtual",
"RootDeviceType": "ebs",
"AmiLaunchIndex": 0
}
]
}
Now lets parse this document using jsawk and see what we can pull out.
Firstly lets pull out the Instances field and its array value
cat run-instances.json | jsawk 'return this.Instances'
As you can see you are able to run JavaScript snippets on the JSON object and pull out parts of the document.
jsawk lets you use all the SpiderMonkey functions in these snippets – reference
That will return the array which Instances contains as shown below
Output
[{"Monitoring":{"State":"disabled"},"PublicDnsName":null,"KernelId":"aki-52a34525","State":{"Code":0,"Name":"pending"},"EbsOptimized":false,"LaunchTime":"2015-05-28T08:34:30.000Z","PrivateIpAddress":"172.31.25.13","ProductCodes":[],"VpcId":"vpc-6bc9110e","StateTransitionReason":null,"InstanceId":"i-ca18d560","ImageId":"ami-1d9eee6a","PrivateDnsName":"ip-172-31-25-13.eu-west-1.compute.internal","KeyName":"orc-server","SecurityGroups":[{"GroupName":"Orc Server","GroupId":"sg-2903204c"}],"ClientToken":null,"SubnetId":"subnet-8024b3e5","InstanceType":"m3.medium","NetworkInterfaces":[{"Status":"in-use","MacAddress":"02:ca:99:88:4d:97","SourceDestCheck":true,"VpcId":"vpc-6bc9110e","Description":null,"NetworkInterfaceId":"eni-6850770c","PrivateIpAddresses":[{"PrivateDnsName":"ip-172-31-25-13.eu-west-1.compute.internal","Primary":true,"PrivateIpAddress":"172.31.25.13"}],"PrivateDnsName":"ip-172-31-25-13.eu-west-1.compute.internal","Attachment":{"Status":"attaching","DeviceIndex":0,"DeleteOnTermination":true,"AttachmentId":"eni-attach-6164ae20","AttachTime":"2015-05-28T08:34:30.000Z"},"Groups":[{"GroupName":"Orc Server","GroupId":"sg-2903204c"}],"SubnetId":"subnet-8024b3e5","OwnerId":"999412538050","PrivateIpAddress":"172.31.25.13"}],"SourceDestCheck":true,"Placement":{"Tenancy":"default","GroupName":null,"AvailabilityZone":"eu-west-1b"},"Hypervisor":"xen","BlockDeviceMappings":[],"Architecture":"x86_64","StateReason":{"Message":"pending","Code":"pending"},"RootDeviceName":"/dev/sda1","VirtualizationType":"paravirtual","RootDeviceType":"ebs","AmiLaunchIndex":1},{"Monitoring":{"State":"disabled"},"PublicDnsName":null,"KernelId":"aki-52a34525","State":{"Code":0,"Name":"pending"},"EbsOptimized":false,"LaunchTime":"2015-05-28T08:34:30.000Z","PrivateIpAddress":"172.31.25.12","ProductCodes":[],"VpcId":"vpc-6bc9110e","StateTransitionReason":null,"InstanceId":"i-cb18d561","ImageId":"ami-1d9eee6a","PrivateDnsName":"ip-172-31-25-12.eu-west-1.compute.internal","KeyName":"orc-server","SecurityGroups":[{"GroupName":"Orc Server","GroupId":"sg-2903204c"}],"ClientToken":null,"SubnetId":"subnet-8024b3e5","InstanceType":"m3.medium","NetworkInterfaces":[{"Status":"in-use","MacAddress":"02:ba:95:c0:5d:a3","SourceDestCheck":true,"VpcId":"vpc-6bc9110e","Description":null,"NetworkInterfaceId":"eni-6950770d","PrivateIpAddresses":[{"PrivateDnsName":"ip-172-31-25-12.eu-west-1.compute.internal","Primary":true,"PrivateIpAddress":"172.31.25.12"}],"PrivateDnsName":"ip-172-31-25-12.eu-west-1.compute.internal","Attachment":{"Status":"attaching","DeviceIndex":0,"DeleteOnTermination":true,"AttachmentId":"eni-attach-4164ae00","AttachTime":"2015-05-28T08:34:30.000Z"},"Groups":[{"GroupName":"Orc Server","GroupId":"sg-2903204c"}],"SubnetId":"subnet-8024b3e5","OwnerId":"999412538050","PrivateIpAddress":"172.31.25.12"}],"SourceDestCheck":true,"Placement":{"Tenancy":"default","GroupName":null,"AvailabilityZone":"eu-west-1b"},"Hypervisor":"xen","BlockDeviceMappings":[],"Architecture":"x86_64","StateReason":{"Message":"pending","Code":"pending"},"RootDeviceName":"/dev/sda1","VirtualizationType":"paravirtual","RootDeviceType":"ebs","AmiLaunchIndex":0}]Now that we have the instances that we just created lets pull out the instanceId for each Instance, so pipe the result back into jsawk and get the InstanceId
cat run-instances.json | jsawk 'return this.Instances' | jsawk 'return this.InstanceId'
The result is an array of the 2 instance Ids of the 2 instances we just created
["i-ca18d560","i-cb18d561"]
But we may want to do something with each of these values, say add a tag to the EC2 instance on creation. What we can do instead of returning the JSON array use a SpiderMonkey method out() and an additional jsawk switch -n to get just the values instead of the JSON array
cat run-instances.json | jsawk 'return this.Instances' | jsawk -n 'return out(this.InstanceId)'
This results in the following output
i-ca18d560i-cb18d561
We can now use standard bash commands to use these values however we want, below is an example of add tags to the instances
The command below assumes you already have the AWS IAM setup and configured with your AWS CLI tools – if you don’t have the tools installed you can do so here
cat run-instances.json | jsawk 'return this.Instances' | jsawk -n 'return out(this.InstanceId)' | /xargs -I{} aws ec2 create-tags --resources {} --tags 'Key=Name,Value=OrcDrones' --region eu-west-1
The result of this command will be tagging your 2 EC2 instances with a Name of OrcDrone
