AWS outbound network access destinations

If you have limited outbound internet access (for example due to using a firewall or proxy), review this content to learn which specific outbound destinations must be available in order to register a CDP environment.

The following list includes general destinations as well as AWS-specific destinations.

General endpoints

Description/Usage

CDP service

Destination

Protocol and Authentication

IP Protocol/Port

Comments

AMPs

Applied ML Prototypes

Machine Learning

https://raw.githubusercontent.com

https://github.com

HTTPS

TCP/443

Files for AMPs are hosted on GitHub.

Cloudera CCMv1

Persistent Control Plane connection

All services

*.ccm.cdp.cloudera.com

44.234.52.96/27

SSH public/private key authentication

TCP/6000-6049

One connection per cluster configured; persistent

Cloudera CCMv2

Persistent Control Plane connection

All services

US-based Control Plane:

*.v2.us-west-1.ccm.cdp.cloudera.com

35.80.24.128/27

35.166.86.177/32

52.36.110.208/32

52.40.165.49/32

EU-based Control Plane:

*.v2.ccm.eu-1.cdp.cloudera.com

3.65.246.128/27

AP-based Control Plane:

*.v2.ccm.ap-1.cdp.cloudera.com

3.26.127.64/27

HTTPS with mutual authentication

TCP/443

Multiple long-lived/persistent connections

Cloudera Databus

Telemetry, billing and metering data

All services

US-based Control Plane:

dbusapi.us-west-1.sigma.altus.cloudera.com

*.s3.amazonaws.com

EU-based Control Plane:

api.eu-1.cdp.cloudera.com

*.s3.amazonaws.com

AP-based Control Plane:

api.ap-1.cdp.cloudera.com

*.s3.amazonaws.com

HTTPS with Cloudera-generated access key for dbus

HTTPS for S3

TCP/443

Regular interval for telemetry, billing, metering services, and used for Workload Manager if enabled. Larger payloads are sent to a Cloudera managed S3 bucket.

Cloudera Manager parcels

Software distribution

Data Hub

Data Lake

Data Engineering

DataFlow

Operational Database

archive.cloudera.com

HTTPS

TCP/443

Cloudera’s public software repository. CDN backed service; IP range not predictable.

Control Plane API

CDP API

US-based Control Plane:

api.us-west-1.cdp.cloudera.com

EU-based Control Plane:

api.eu-1.cdp.cloudera.com

AP-based Control Plane:

api.ap-1.cdp.cloudera.com

HTTPS with Cloudera-generated access key

TCP/443

Cloudera’s control plane REST API.

Control Plane API

Data Engineering

DataFlow

Machine Learning

api.us-west-1.cdp.cloudera.com

HTTPS with Cloudera-generated access key

TCP/443

Cloudera’s control plane REST API.

Docker Images

Software Distribution

Data Engineering

DataFlow

Machine Learning

container.repository.cloudera.com

docker.repository.cloudera.com

HTTPS

TCP/443

Cloudera’s public docker registry. CDN backed service; IP range not predictable.

Docker Images

Software Distribution

Data Engineering

DataFlow

Data Warehouse

container.repo.cloudera.com

*.s3.<DOCKER-REGISTRY-REGION>.amazonaws.com

s3-r-w.<DOCKER-REGISTRY-REGION>.amazonaws.com

*.execute-api.<DOCKER-REGISTRY-REGION>.amazonaws.com

Additionally, the following are required only for old/existing Data Warehouse environments:

auth.docker.io*

cloudera-docker-dev.jfrog.io*

docker-images-prod.s3.amazonaws.com*

gcr.io*

k8s.gcr.io*

quay-registry.s3.amazonaws.com*

quay.io*

quayio-production-s3.s3.amazonaws.com*

docker.io*

production.cloudflare.docker.com*

storage.googleapis.com*

HTTPS

TCP/443

Moved to container.repo.cloudera.com

container.repo.cloudera.com uses ECR which requires S3 URLs.

Identity Provider Discovery

DataFlow

consoleauth.us-west-1.core.altus.cloudera.com

HTTPS

TCP/443

None

Draft comment:
Add this comment for the September 2021 DF GA: Necessary to support access token verification using the CDP CLI.

Public Signing Key Retrieval

Data Engineering

DataFlow

consoleauth.altus.cloudera.com

HTTPS

TCP/443

Required to allow authentication to CDE virtual Cluster using a CDP Access Key.

SQL Stream Builder PostgreSQL driver install

Data Hub: Streaming Analytics clusters

pypi.org

HTTPS

TCP/443

SQL Stream Builder depends on the python3 PostgreSQL driver.

This is only required for Runtime versions 7.2.11, 7.2.12 and 7.2.13.

Draft comment:

When CCMv2 is GA, the endpoint for all other regions besides us-west-1 SHOULD be (verify with Enis): *.v2.<control-plane-region>.ccm.cdp.cloudera.com

AWS-specific endpoints

Description/Usage

CDP service

Destination

Protocol and Authentication

IP Protocol/Port

Comments

AWS STS

Data Lake

sts.amazonaws.com

sts.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

CDP 7.1.1+ required before can be made internal with VPC endpoints.

AWS S3

All services

*.s3.amazonaws.com

*.s3.[***WORKLOAD-REGION***].amazonaws.com

s3.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

*.s3.[***WORKLOAD-REGION***].amazonaws.com is VPC internal.

*.s3.amazonaws.com and s3.amazonaws.com can be made internal with VPC endpoints.

AWS DynamoDB

All services

dynamodb.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

Can be made internal with VPC endpoints.

AWS RDS

Data Lake

Data Hub

Data Engineering

DataFlow

*.*.rds.amazonaws.com

JDBC / Postgres binary protocol / MySQL

TCP 5432 / 3306

VPC Internal.

Only Data Engineering uses MySQL and requires port 3306 to be open.

AWS EC

DataFlow

Data Warehouse

Machine Learning

api.ecr.[***WORKLOAD-REGION***].amazonaws.com

*.dkr.ecr.[***WORKLOAD-REGION***].amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

VPC Internal.

AWS EC2

DataFlow

Data Warehouse

Machine Learning

Operational Database

ec2.[***WORKLOAD-REGION***].amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

VPC Internal.

AWS EKS

Data Engineering

DataFlow

Data Warehouse

Machine Learning

eks.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

AWS does not support EKS VPC endpoints at this time.

AWS Cloudformation

DataFlow

Data Warehouse

Machine Learning

cloudformation.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

Can be made internal with VPC endpoints.

AWS Autoscaling

Data Engineering

DataFlow

Data Warehouse

Machine Learning

autoscaling.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

Can be made internal with VPC endpoints.

AWS EFS

Data Engineering

Data Warehouse

Machine Learning

elasticfilesystem.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

Can be made internal with VPC endpoints.

AWS EKS k8s cluster api

Data Warehouse

UNIQUEID.*.eks.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

Optional for new clusters.

AWS ELB

Data Engineering

Data Warehouse

elasticloadbalancing.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

Can be made internal with VPC endpoints.

AWS RDS API

Data Warehouse

rds.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

AWS does not support RDS API VPC endpoints at this time. This requirement is under further evaluation.

Data Warehouse uses Amazon RDS for PostgreSQL.

AWS Service Quotas

Data Warehouse

servicequotas.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

AWS does not support Service Quota via VPC endpoints. Used to check limits and warn prior to hitting the limits.

AWS Price List Service

DataFlow

Data Warehouse

pricing.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

AWS Price List Service uses us-east-1 or ap-south-1 as the region.

Flow definitions storage

DataFlow

s3.*.amazonaws.com

HTTPS (one way)

IAM authentication

TCP/443

Currently us-west-1 is the only supported region.

The host can be made internal with VPC endpoints.