Handling JSON in BASH

< Back to blog

I found a really cool github project which extends the bash utility awk to handle JSON objects in bash. Enter Jsawk

 

Installation

Installing the utility is pretty easy, we pretty much grab it from github

cd ~curl -L http://github.com/micha/jsawk/raw/master/jsawk > jsawk

Next we make it executable and put it somewhere we can access it

chmod 755 jsawk && mv jsawk ~/bin/

 

Dependencies

One dependency which jsawk has if you are running this on Mac is a JS binary. OSX doesn’t seem to have one already installed so lets go and grab one. Enter SpiderMonkey.

SpiderMonkey is a JavaScript engine used in FireFox, so lets download it and put it somewhere sensible

cd ~/binwget http://web.stanford.edu/class/cs242/spidermonkey/js-osx.zipunzip js-osx.zip

That should give you an executable utility called js, now we can tell jsawk about it with the following switch

jsawk -j <path to spidermonkey js>

 

Alias

To make things easy lets add an alias so that we dont need to keep adding this switch when we want to use the utility

vim ~/.bash_profile #addalias jsawk='~/bin/jsawk -j ~/bin/js'

Now just running jsawk will work without any problems

 

Usage

Here are some examples of usage.

Below is an example JSON document from an output of AWS CLI command to create 2 EC2 instances in AWS

run-instances.json
{
    "OwnerId": "999412538050",
    "ReservationId": "r-3332b3d7",
    "Groups": [],
    "Instances": [
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": null,
            "KernelId": "aki-52a34525",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2015-05-28T08:34:30.000Z",
            "PrivateIpAddress": "172.31.25.13",
            "ProductCodes": [],
            "VpcId": "vpc-6bc9110e",
            "StateTransitionReason": null,
            "InstanceId": "i-ca18d560",
            "ImageId": "ami-1d9eee6a",
            "PrivateDnsName": "ip-172-31-25-13.eu-west-1.compute.internal",
            "KeyName": "orc-server",
            "SecurityGroups": [
                {
                    "GroupName": "Orc Server",
                    "GroupId": "sg-2903204c"
                }
            ],
            "ClientToken": null,
            "SubnetId": "subnet-8024b3e5",
            "InstanceType": "m3.medium",
            "NetworkInterfaces": [
                {
                    "Status": "in-use",
                    "MacAddress": "02:ca:99:88:4d:97",
                    "SourceDestCheck": true,
                    "VpcId": "vpc-6bc9110e",
                    "Description": null,
                    "NetworkInterfaceId": "eni-6850770c",
                    "PrivateIpAddresses": [
                        {
                            "PrivateDnsName": "ip-172-31-25-13.eu-west-1.compute.internal",
                            "Primary": true,
                            "PrivateIpAddress": "172.31.25.13"
                        }
                    ],
                    "PrivateDnsName": "ip-172-31-25-13.eu-west-1.compute.internal",
                    "Attachment": {
                        "Status": "attaching",
                        "DeviceIndex": 0,
                        "DeleteOnTermination": true,
                        "AttachmentId": "eni-attach-6164ae20",
                        "AttachTime": "2015-05-28T08:34:30.000Z"
                    },
                    "Groups": [
                        {
                            "GroupName": "Orc Server",
                            "GroupId": "sg-2903204c"
                        }
                    ],
                    "SubnetId": "subnet-8024b3e5",
                    "OwnerId": "999412538050",
                    "PrivateIpAddress": "172.31.25.13"
                }
            ],
            "SourceDestCheck": true,
            "Placement": {
                "Tenancy": "default",
                "GroupName": null,
                "AvailabilityZone": "eu-west-1b"
            },
            "Hypervisor": "xen",
            "BlockDeviceMappings": [],
            "Architecture": "x86_64",
            "StateReason": {
                "Message": "pending",
                "Code": "pending"
            },
            "RootDeviceName": "/dev/sda1",
            "VirtualizationType": "paravirtual",
            "RootDeviceType": "ebs",
            "AmiLaunchIndex": 1
        },
        {
            "Monitoring": {
                "State": "disabled"
            },
            "PublicDnsName": null,
            "KernelId": "aki-52a34525",
            "State": {
                "Code": 0,
                "Name": "pending"
            },
            "EbsOptimized": false,
            "LaunchTime": "2015-05-28T08:34:30.000Z",
            "PrivateIpAddress": "172.31.25.12",
            "ProductCodes": [],
            "VpcId": "vpc-6bc9110e",
            "StateTransitionReason": null,
            "InstanceId": "i-cb18d561",
            "ImageId": "ami-1d9eee6a",
            "PrivateDnsName": "ip-172-31-25-12.eu-west-1.compute.internal",
            "KeyName": "orc-server",
            "SecurityGroups": [
                {
                    "GroupName": "Orc Server",
                    "GroupId": "sg-2903204c"
                }
            ],
            "ClientToken": null,
            "SubnetId": "subnet-8024b3e5",
            "InstanceType": "m3.medium",
            "NetworkInterfaces": [
                {
                    "Status": "in-use",
                    "MacAddress": "02:ba:95:c0:5d:a3",
                    "SourceDestCheck": true,
                    "VpcId": "vpc-6bc9110e",
                    "Description": null,
                    "NetworkInterfaceId": "eni-6950770d",
                    "PrivateIpAddresses": [
                        {
                            "PrivateDnsName": "ip-172-31-25-12.eu-west-1.compute.internal",
                            "Primary": true,
                            "PrivateIpAddress": "172.31.25.12"
                        }
                    ],
                    "PrivateDnsName": "ip-172-31-25-12.eu-west-1.compute.internal",
                    "Attachment": {
                        "Status": "attaching",
                        "DeviceIndex": 0,
                        "DeleteOnTermination": true,
                        "AttachmentId": "eni-attach-4164ae00",
                        "AttachTime": "2015-05-28T08:34:30.000Z"
                    },
                    "Groups": [
                        {
                            "GroupName": "Orc Server",
                            "GroupId": "sg-2903204c"
                        }
                    ],
                    "SubnetId": "subnet-8024b3e5",
                    "OwnerId": "999412538050",
                    "PrivateIpAddress": "172.31.25.12"
                }
            ],
            "SourceDestCheck": true,
            "Placement": {
                "Tenancy": "default",
                "GroupName": null,
                "AvailabilityZone": "eu-west-1b"
            },
            "Hypervisor": "xen",
            "BlockDeviceMappings": [],
            "Architecture": "x86_64",
            "StateReason": {
                "Message": "pending",
                "Code": "pending"
            },
            "RootDeviceName": "/dev/sda1",
            "VirtualizationType": "paravirtual",
            "RootDeviceType": "ebs",
            "AmiLaunchIndex": 0
        }
    ]
}

Now lets parse this document using jsawk and see what we can pull out.

Firstly lets pull out the Instances field and its array value

cat run-instances.json | jsawk 'return this.Instances'

As you can see you are able to run JavaScript snippets on the JSON object and pull out parts of the document.

jsawk lets you use all the SpiderMonkey functions in these snippets – reference

That will return the array which Instances contains as shown below

Output
[{"Monitoring":{"State":"disabled"},"PublicDnsName":null,"KernelId":"aki-52a34525","State":{"Code":0,"Name":"pending"},"EbsOptimized":false,"LaunchTime":"2015-05-28T08:34:30.000Z","PrivateIpAddress":"172.31.25.13","ProductCodes":[],"VpcId":"vpc-6bc9110e","StateTransitionReason":null,"InstanceId":"i-ca18d560","ImageId":"ami-1d9eee6a","PrivateDnsName":"ip-172-31-25-13.eu-west-1.compute.internal","KeyName":"orc-server","SecurityGroups":[{"GroupName":"Orc Server","GroupId":"sg-2903204c"}],"ClientToken":null,"SubnetId":"subnet-8024b3e5","InstanceType":"m3.medium","NetworkInterfaces":[{"Status":"in-use","MacAddress":"02:ca:99:88:4d:97","SourceDestCheck":true,"VpcId":"vpc-6bc9110e","Description":null,"NetworkInterfaceId":"eni-6850770c","PrivateIpAddresses":[{"PrivateDnsName":"ip-172-31-25-13.eu-west-1.compute.internal","Primary":true,"PrivateIpAddress":"172.31.25.13"}],"PrivateDnsName":"ip-172-31-25-13.eu-west-1.compute.internal","Attachment":{"Status":"attaching","DeviceIndex":0,"DeleteOnTermination":true,"AttachmentId":"eni-attach-6164ae20","AttachTime":"2015-05-28T08:34:30.000Z"},"Groups":[{"GroupName":"Orc Server","GroupId":"sg-2903204c"}],"SubnetId":"subnet-8024b3e5","OwnerId":"999412538050","PrivateIpAddress":"172.31.25.13"}],"SourceDestCheck":true,"Placement":{"Tenancy":"default","GroupName":null,"AvailabilityZone":"eu-west-1b"},"Hypervisor":"xen","BlockDeviceMappings":[],"Architecture":"x86_64","StateReason":{"Message":"pending","Code":"pending"},"RootDeviceName":"/dev/sda1","VirtualizationType":"paravirtual","RootDeviceType":"ebs","AmiLaunchIndex":1},{"Monitoring":{"State":"disabled"},"PublicDnsName":null,"KernelId":"aki-52a34525","State":{"Code":0,"Name":"pending"},"EbsOptimized":false,"LaunchTime":"2015-05-28T08:34:30.000Z","PrivateIpAddress":"172.31.25.12","ProductCodes":[],"VpcId":"vpc-6bc9110e","StateTransitionReason":null,"InstanceId":"i-cb18d561","ImageId":"ami-1d9eee6a","PrivateDnsName":"ip-172-31-25-12.eu-west-1.compute.internal","KeyName":"orc-server","SecurityGroups":[{"GroupName":"Orc Server","GroupId":"sg-2903204c"}],"ClientToken":null,"SubnetId":"subnet-8024b3e5","InstanceType":"m3.medium","NetworkInterfaces":[{"Status":"in-use","MacAddress":"02:ba:95:c0:5d:a3","SourceDestCheck":true,"VpcId":"vpc-6bc9110e","Description":null,"NetworkInterfaceId":"eni-6950770d","PrivateIpAddresses":[{"PrivateDnsName":"ip-172-31-25-12.eu-west-1.compute.internal","Primary":true,"PrivateIpAddress":"172.31.25.12"}],"PrivateDnsName":"ip-172-31-25-12.eu-west-1.compute.internal","Attachment":{"Status":"attaching","DeviceIndex":0,"DeleteOnTermination":true,"AttachmentId":"eni-attach-4164ae00","AttachTime":"2015-05-28T08:34:30.000Z"},"Groups":[{"GroupName":"Orc Server","GroupId":"sg-2903204c"}],"SubnetId":"subnet-8024b3e5","OwnerId":"999412538050","PrivateIpAddress":"172.31.25.12"}],"SourceDestCheck":true,"Placement":{"Tenancy":"default","GroupName":null,"AvailabilityZone":"eu-west-1b"},"Hypervisor":"xen","BlockDeviceMappings":[],"Architecture":"x86_64","StateReason":{"Message":"pending","Code":"pending"},"RootDeviceName":"/dev/sda1","VirtualizationType":"paravirtual","RootDeviceType":"ebs","AmiLaunchIndex":0}]

Now that we have the instances that we just created lets pull out the instanceId for each Instance, so pipe the result back into jsawk and get the InstanceId

cat run-instances.json | jsawk 'return this.Instances' | jsawk 'return this.InstanceId'

The result is an array of the 2 instance Ids of the 2 instances we just created

["i-ca18d560","i-cb18d561"]

But we may want to do something with each of these values, say add a tag to the EC2 instance on creation. What we can do instead of returning the JSON array use a SpiderMonkey method out() and an additional jsawk switch -n to get just the values instead of the JSON array

cat run-instances.json | jsawk 'return this.Instances' | jsawk -n 'return out(this.InstanceId)'

This results in the following output

i-ca18d560i-cb18d561

We can now use standard bash commands to use these values however we want, below is an example of add tags to the instances

The command below assumes you already have the AWS IAM setup and configured with your AWS CLI tools – if you don’t have the tools installed you can do so here

cat run-instances.json | jsawk 'return this.Instances' | jsawk -n 'return out(this.InstanceId)' | /xargs -I{} aws ec2 create-tags --resources {} --tags 'Key=Name,Value=OrcDrones' --region eu-west-1

The result of this command will be tagging your 2 EC2 instances with a Name of OrcDrone

Published by Andrew Kew | See all posts by Andrew Kew