jq usage memo - Welcome to new things

Shell scripts use the jq command to retrieve data from JSON.

I often forget how to use it, so here is a memo for my own use.

Basic

Basic operation "takes input from standard input, filters it, and outputs it to standard output."
. represents the input value
Filters are connected by pipes (|)
Wrap filters in single quotes
If you use double quotation marks in the filter, escape them with a backslash and enter

Other

Can also be used simply as a JSON formatting tool (jq '.')
Can generate JSON strings
Test Site
- A playground for jq

array (programming, programing)

Extract by specifying the index with . [index].
Indexes can also use negative values.
Multiple indexes can be specified.
[] expands the entire array
To convert the expanded array back into an array, enclose the entire array in []
- Cannot go from an expanded array back to an array via pipe

# Input
[ "a", 123, "b" ]

##########
# Specify array elements by index
# (Specify array elements by index)
jq '.[0]'
# 結果 (result)
"a"

##########
# Use negative values for indexes
# (Use negative values for indexes)
jq '.[-1]'
# 結果 (result)
"b"

##########
# Specify multiple indexes
# (Specify multiple indexes)
jq '.[1, 2]'
# 結果 (result)
123
"b"

##########
# Expand the array
# (Expand the array)
jq '.[]'
# 結果 (result)
"a"
123
"b"

##########
# Returns the expanded array to an array.
# (Returns the expanded array to an array.)
jq '[.[]]'
# 結果 (result)
[
  "a",
  123,
  "b"
]

##########
# Once an array is expanded, it cannot be returned to an array later.
# (Once an array is expanded, it cannot be returned to an array later.)
jq '.[] | [.]'
# 結果 (result)
[ "a" ]
[ 123 ]
[ "b" ]

object

Extracting the value of an object key with .key
Can also be specified with object["key"] as in JavaScript
- Especially useful when key names contain special characters or start with a number and cannot be specified with .key.
Using [] for an object, the values of all keys are retrieved
To use the retrieved value as a key for the next object, enclose the value in ().

# Input
{
    "a": "A",
    "b": 123,
    "123c": "TEST"
}

##########
# Retrieve the value of an object's key
# (Retrieve the value of an object's key)
jq '.b'
# 結果 (result)
123

##########
# Specify key in [].
# (Specify key in [].)
jq '.["123c"]'
# 結果 (result)
"TEST"

##########
# [] gets the value of all keys of the object
# ([] gets the value of all keys of the object)
jq '.[]'
# 結果 (result)
"A"
123
"TEST"

##########
# Create an object with the obtained value as the key name
# (Create an object with the obtained value as the key name)
jq '{(.b):.a}'
# 結果 (result)
{
    "123": "A"
}

Array and object generation

A new array can be generated with [].
New objects can be created with {}

# Input
{
  "a": "A",
  "b": 123
}

##########
# Generate array
# (Generate array)
jq '[.a, .b]'
# 結果 (result)
[
  "A",
  123
]

##########
# Generating Objects
# (Generating Objects)
jq '{aa:.a, bb:.b}'
# 結果 (result)
{
  "aa": "A",
  "bb": 123
}

pipe

. can be used to write key by connecting key.
Processes can be connected with |

# Input
{
    "a": {
        "b":["test1", "test2", "test3"]
    }
}

##########
# Keys can be concatenated with "." can be concatenated with
# (Keys can be concatenated with "." can be concatenated with)
jq '.a.b[1]'
# 結果 (result)
"test2"

##########
# Piping with "|".
# (Piping with "|".)
jq '.a|.b|.[1]'
# 結果 (result)
"test2"

##########
# |" is equivalent to piping jq
# ("|" is equivalent to piping jq)
jq '.a' | jq '.b' | jq '.[1]'
# 結果 (result)
"test2"

Functions and Assignments

|=

| is a pipe
Filter the left side, pipe in | and move to the right side. Then do the operations on the right side, and update the value with the result back to the filter on the left side.

# Input
[1, 2, 3]

##########
# Update the value of index 0 by computing
# (Update the value of index 0 by computing)
jq '.[0]|=.*100'
# 結果 (result)
[
    100,
    2,
    3
]

##########
# Operate and update each value in the array
# (Operate and update each value in the array)
jq '.[]|=.*100'
# 結果 (result)
[
    100,
    200,
    300
]

# Input
{
    "a":1,
    "b":2,
    "c":3
}

##########
# Only .a is calculated and updated
# (Only .a is calculated and updated)
jq '.a|=.*100'
# 結果 (result)
{
    "a":100,
    "b":2,
    "c":3
}

##########
# Operate and update object values
# (Operate and update object values)
jq '.[]|=.*100'
# 結果 (result)
{
    "a":100,
    "b":200,
    "c":300
}

=

Substitute

# Input
{"a": {"b": 10}, "b": 20}

##########
# .a,.b refer to the same hierarchy
# (.a,.b refer to the same hierarchy)
jq '.a = .b'
# 結果 (result)
{
  "a": 20,
  "b": 20
}

##########
# .a.b is a child of .a
# (.a.b is a child of .a)
jq '.a = .a.b'
# 結果 (result)
{
  "a": 10,
  "b": 20
}

##########
# exp
# Since "|" is a pipe, the right ". is in the left filter (.a)
# (Since "|" is a pipe, the right ". is in the left filter (.a))
jq '.a |= .b'
# 結果 (result)
{
  "a": 10,
  "b": 20
}

keys

Returns an array of object keys
For arrays, the index is returned.

# Input
{
    "a": "TEST",
    "b": 123
}

##########
# Returns an array of object keys with keys
# (Returns an array of object keys with keys)
jq 'keys'
# 結果 (result)
[
    "a",
    "b"
]

# Input
[ "a", "b", "c" ]

##########
# Returns the index of an array with keys
# (Returns the index of an array with keys)
jq 'keys'
# 結果 (result)
[
    0,
    1,
    2
]

map(x)

Same as [.[]|x]
Expand the array, perform operations with each value, and return the result as an array
- This is combined into one function
The end result will be an array

# Input
[ 1, 2, 3 ]

##########
# Arithmetic operations with each element of an array
# (Arithmetic operations with each element of an array)
jq 'map(.*100)'
# 結果 (result)
[
    100,
    200,
    300
]

# Input
{
    "a":1,
    "b":2,
    "c":3
}

##########
# Operate with each element of an object
# (Operate with each element of an object)
jq 'map(.*100)'
# 結果 (result)
[
    100,
    200,
    300
]

map_values(x)

Same as .[]|=x
Operate with each value of the object and update with the result
The end result is an object if the input is an object, or an array if it is an array

# Input
[ 1, 2, 3 ]

##########
# Arithmetic operations with each element of an array
# (Arithmetic operations with each element of an array)
jq 'map_values(.*100)'
# 結果 (result)
[
    100,
    200,
    300
]

# Input
{
    "a":1,
    "b":2,
    "c":3
}

##########
# Operate with each element of an object
# (Operate with each element of an object)
jq 'map_values(.*100)'
# 結果 (result)
{
    "a":100,
    "b":200,
    "c":300
}

select(x)

Leave those that meet the requirements
Commonly used to retrieve matching items from an array, such as map(select(x)), .[]|select(x), etc.

# Input
[1,5,3,0,7]

##########
# Filtering Arrays
# (Filtering Arrays)
jq 'map(select(. >= 2))'
# 結果 (result)
[
    5,
    3,
    7
]

##########
# Expand the array and keep the ones that match the criteria from each element.
# (Expand the array and keep the ones that match the criteria from each element.)
jq '.[]|select(. >= 2)'
# 結果 (result)
5
3
7

# Input
[
    {"a":1, "b":100},
    {"a":5, "b":500},
    {"a":3, "b":300},
    {"a":0, "b":0},
    {"a":7, "b":700}
]

##########
# Filtering Arrays
# (Filtering Arrays)
jq 'map(select(.a >= 2))'
# 結果 (result)
[
    {"a":5, "b":500},
    {"a":3, "b":300},
    {"a":7, "b":700}
]

##########
# Expand the array and keep the ones that match the criteria from each element.
# (Expand the array and keep the ones that match the criteria from each element.)
jq '.[]|select(.a >= 2)'
# 結果 (result)
{"a":5, "b":500},
{"a":3, "b":300},
{"a":7, "b":700}

to_entries

Make an object into an array of key/value pairs
If an array is passed, key becomes the index of the array

# Input
{"a":1, "b":"TEST"}

##########
# turn an object into an array of ``key/value`` pairs
# (turn an object into an array of ``key/value`` pairs)
jq 'to_entries'
# 結果 (result)
[
  {
    "key": "a",
    "value": 1
  },
  {
    "key": "b",
    "value": "TEST"
  }
]

# Input
[{"a":1, "b":"TEST"}]

##########
# Make an array of ``index/value`` pairs
# (Make an array of ``index/value`` pairs)
jq 'to_entries'
# 結果 (result)
[
  {
    "key": 0,
    "value": {
      "a": 1,
      "b": "TEST"
    }
  }
]

tojson

JSON to JSON string

# Input
[
  {"a":1, "b":"TEST1"},
  {"a":2, "b":"TEST2"}
]

##########
# JSON to JSON string
# (JSON to JSON string)
jq '.[]|tojson'
# 結果 (result)
"{\"a\":1,\"b\":\"TEST1\"}"
"{\"a\":2,\"b\":\"TEST2\"}"

@csv

csv output
Describe lines to be output in csv as an array
Exclude double quotation marks with the -r option.
Header output is not available.

# Input
[
  { "a": "A", "b": 1, "c": "TEST1"},
  { "a": "B", "b": 2, "c": "TEST2"},
  { "a": "C", "b": 3, "c": "TEST3"}
]

##########
# CSV output
# (CSV output)
jq -r '.[]|[.a, .b, .c]|@csv'
# 結果 (result)
"A",1,"TEST1"
"B",2,"TEST2"
"C",3,"TEST3"

variable

The as $var can be used to memorize a value in a variable.

# Input
{
  "userID":123,
  "userInfo":{
    "name":"TEST"
  },
  "data":[
    {"a":"A", "b":1},
    {"a":"B", "b":2},
    {"a":"C", "b":3}
  ]
}

##########
# Use variables to use values elsewhere
# (Use variables to use values elsewhere)
jq '.userID as $userID|.userInfo.name as $name|.data[]|{userID:$userID, name:$name, a:.a, b:.b}'
# 結果 (result)
{
  "userID": 123,
  "name": "TEST",
  "a": "A",
  "b": 1
}
{
  "userID": 123,
  "name": "TEST",
  "a": "B",
  "b": 2
}
{
  "userID": 123,
  "name": "TEST",
  "a": "C",
  "b": 3
}

variable external assignment

Variables can be assigned from the outside.
Assign the number --argjson var_name value.
Assign the character --arg var_name value.

# Input
{
  "aa":12,
  "bb":"AB"
}

##########
# Assign values to variables from the outside and use them internally
# (Assign values to variables from the outside and use them internally)
jq --argjson cc 34 --arg dd "CD" '{a:.aa, b:.bb, c:$cc, d:$dd"}'
# 結果 (result)
{
  "a": 12,
  "b": "AB",
  "c": 34,
  "d": "CD"
}

option

-c
- Output one JSON in one line
-r
- Do not display double quotation marks in strings
-n
- Does not take input; use when creating your own JSON.

Shell script to create JSON data

a=12
b=AB

json=$(cat <<__JSON__
{
  "a": $a,
  "b": "$b"
}
__JSON__
)

echo $json | jq .
# 結果 (result)
# {
#     "a": 12,
#     "b": "AB"
# }

Impressions, etc.

The . is like the this of the programming language.

Arrays" and "expanded arrays" are two different things, although they are similar and easily confused.

Arrays" are output as a single array, whereas "expanded arrays" split the output into separate elements.