redshift spectrum vs athena vs s3 select

#REDSHIFT SPECTRUM VS ATHENA VS S3 SELECT FULL#

If you mostly need to access the data just one way (e.g. The latter key schema would be the other way around.

#REDSHIFT SPECTRUM VS ATHENA VS S3 SELECT FULL#

If you wanted to search only by type x but don't know the date, it would require a full bucket scan. The former key structure allows you to limit the amount of data to be scanned by date or date and type but not type alone. Let's assume you have event type and time in events. Athena directly uses S3 key structure to limit the amount of data to be scanned. Which one to use depends on your data and access patterns. This is dependent on your use case (volumes of data and types of query that you need to run). You MAY NOT need to convert to parquet, if you use the right partitioning structure (s3 folders) and gzip the data then Athena/spectrum then performance can be good enough without the complexity of conversion to parquet. Map the parquet in to Athena and Redshift Spectrum.ĮVENTS -> STORE IT IN S3 -> HIVE to convert to parquet -> Use directly in AthenaĮVENTS -> STORE IT IN S3 -> HIVE to convert to parquet -> Use directly in Redshift using Redshift Spectrum AWS Glue Crawler can be a great way to create the metadata needed to.

Redshift to bring the data in if you need to. View against your parquet tables, then if necessary a CTAS within If you want to use Redshift, then use Redshift spectrum to set up a.

Athena is great when used against parquet, so you don't need to use.

You don't have to use spark to convert to parquet, there is also the option of using hive.

Spark-Redshift works fine but is a complex solution.

Redshift spectrum vs athena vs s3 select