Aggregation of enterprise data and exporting large datasets to third parties
Assume a large organization with multiple applications/systems that may or may not be connected. All systems are currently on-prem. There are requirements to aggregate data from various sources (internal databases like DB2, MariaDB, PostgreSQL), export data to large data files (currently mostly XML) and send them to third parties in a secure fashion (currently SFTP). The legacy system responsible for doing this is at the end of its life.
If I wanted to replace the legacy system with a cloud solution,
1. What kind a data store would be best, a data lake (or some other HDFS-based storage), a data warehouse (Stretch database?), CosmosDB, or something else?
2. What options are there for transfering data from on-prem OLTP databases to the cloud storage? I would prefer to avoid hard-to-maintain ETL-processes. Some kind of change feed would be preferred.
3. What options do I have for sharing the data files with third party partners from Azure storage? The partners don’t necessarily have an Azure subscription so Azure Data Share isn’t always an option?
Assume a large organization with multiple applications/systems that may or may not be connected. All systems are currently on-prem. There are requirements to aggregate data from various sources (internal databases like DB2, MariaDB, PostgreSQL), export data to large data files (currently mostly XML) and send them to third parties in a secure fashion (currently SFTP). The legacy system responsible for doing this is at the end of its life. If I wanted to replace the legacy system with a cloud solution, 1. What kind a data store would be best, a data lake (or some other HDFS-based storage), a data warehouse (Stretch database?), CosmosDB, or something else?2. What options are there for transfering data from on-prem OLTP databases to the cloud storage? I would prefer to avoid hard-to-maintain ETL-processes. Some kind of change feed would be preferred.3. What options do I have for sharing the data files with third party partners from Azure storage? The partners don’t necessarily have an Azure subscription so Azure Data Share isn’t always an option? Read More
