I have 14 tables on a client server that need to be fetched using REST API calls. Each table contains between 71 and 800,000 rows, but the page size for fetching data is limited to 100 rows. The REST API does not provide a nextPage link; instead, it uses the parameter “startRow” to specify the starting row for each page. For example, setting startRow=101 fetches rows 101-200.

Our goal is to copy the tables from the REST API connector to ADLS storage as .json files and then transfer the data to a SQL Server. Each table should be stored as a single .json file in ADLS, despite being fetched in multiple pages. Thus, we need 14 .json files, one for each table, in the ADLS storage folder.

Could someone guide me on how to configure the copy data activity’s source and sink to achieve this? Any leads would be greatly appreciated.

#ADF #copyactivity #pagination

​I have 14 tables on a client server that need to be fetched using REST API calls. Each table contains between 71 and 800,000 rows, but the page size for fetching data is limited to 100 rows. The REST API does not provide a nextPage link; instead, it uses the parameter “startRow” to specify the starting row for each page. For example, setting startRow=101 fetches rows 101-200.Our goal is to copy the tables from the REST API connector to ADLS storage as .json files and then transfer the data to a SQL Server. Each table should be stored as a single .json file in ADLS, despite being fetched in multiple pages. Thus, we need 14 .json files, one for each table, in the ADLS storage folder.Could someone guide me on how to configure the copy data activity’s source and sink to achieve this? Any leads would be greatly appreciated.#ADF #copyactivity #pagination  Read More

​