Flattening nested JSON values in a dataflow with varying keys.
We are using Azrue DevOps REST API calls to return JSON files and storing them in blob. Then we perform a dataflow to transform the data. The issue is a portion of the JSON being stored in blob has varying keys.
When we specify the columns to map in a Select action, we are selecting specifically one of the varying keys from a list of options. But need to map ALL of these – we cannot manually specify these because the data source is so large.
We cannot implement a standard name for this section of the JSON. A wildcard for { } would work ideally but is not supported. We do not care what the keys are, just the contents (id, name).
Select Action:
Source Column:
resources.pipelines.{src-release}.pipeline.id
resources.pipelines.{src-release}.pipeline.name
resources.pipelines.{build }.pipeline.id
resources.pipelines.{build }.pipeline.name
Mapping Name as:
‘pipelineID’
‘pipelineName’
Below is a JSON snippet which highlights the key from the source JSON
Example of Select action mapping – each key shows as its own dropdown:
We are using Azrue DevOps REST API calls to return JSON files and storing them in blob. Then we perform a dataflow to transform the data. The issue is a portion of the JSON being stored in blob has varying keys. When we specify the columns to map in a Select action, we are selecting specifically one of the varying keys from a list of options. But need to map ALL of these – we cannot manually specify these because the data source is so large. We cannot implement a standard name for this section of the JSON. A wildcard for { } would work ideally but is not supported. We do not care what the keys are, just the contents (id, name). Select Action:Source Column:resources.pipelines.{src-release}.pipeline.idresources.pipelines.{src-release}.pipeline.nameresources.pipelines.{build }.pipeline.idresources.pipelines.{build }.pipeline.name Mapping Name as:‘pipelineID’‘pipelineName’Below is a JSON snippet which highlights the key from the source JSON Example of Select action mapping – each key shows as its own dropdown: Read More
