vuthanhdatt's Second Brain

Deploy Dagster

1 min read

dagster k8s

Set up cluster

Test with minikube cluster first

Docker image with code

Using availabe example docker image in dagster/user-code-example Tags | Docker Hub

S3 service with minio

export AWS_ACCESS_KEY_ID="minio"
export AWS_SECRET_ACCESS_KEY="minioadmin"
kubectl create secret generic dagster-aws-access-key-id --from-literal=AWS_ACCESS_KEY_ID=minio
kubectl create secret generic dagster-aws-secret-access-key --from-literal=AWS_SECRET_ACCESS_KEY=minioadmin

Helm

helm repo add dagster https://dagster-io.github.io/helm

helm search repo dagster

To use new releases of the Dagster Helm chart, we’ll need to update our chart repository cache:

helm repo update

dagster-user-deployments:
  enabled: true
  deployments:
    - name: "k8s-example-user-code-1"
      image:
        repository: "docker.io/dagster/user-code-example"
        tag: latest
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "/example_project/example_repo/repo.py"
      port: 3030
      envSecrets:
        - name: dagster-aws-access-key-id
        - name: dagster-aws-secret-access-key
 
runLauncher:
  type: K8sRunLauncher
  config:
    k8sRunLauncher:
      envSecrets:
        - name: dagster-aws-access-key-id
        - name: dagster-aws-secret-access-key

Graph View