airflow 설치 / 설정

도커로 설치하니 메모리를 너무 많이 사용해서 간단 설치하기로 함.

1. 따라하기

파이썬 설치.. 안될 경우 추가. 

1. 파이썬 설치. 
$$ sudo apt-cache policy python3.8
python3.8:
  Installed: (none)
  Candidate: (none)
  Version table:

$$ sudo apt install software-properties-common
$$ sudo add-apt-repository ppa:deadsnakes/ppa
$$ sudo apt install python3.8
$$ python -V
Python 3.8.6
$$


2. airflow 설치
참조 : https://airflow.apache.org/docs/apache-airflow/stable/start.html

$$ mkdir airflow;cd airflow

python 가상환경을 생성합니다.
$$ python -m venv .venv

활성화
$$ source .venv/bin/activate
(.venv) $$

$$ export AIRFLOW_HOME=/home/zeppelin/airflow
$$ pip3 install "apache-airflow==2.3.4" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.3.4/constraints-3.8.txt"

(.venv) $$ pip3 check
No broken requirements found.
(.venv) $$ python3 -m airflow
usage: airflow [-h] GROUP_OR_COMMAND ...

** airflow.cfg 수정 : base_url, web_server_port, endpoint_url, load_examples

$$ airflow standalone
...
Login with username: admin  password: KhnZydnsqN76t4Xy

 

 

 

 

 

 

 

https://airflow.apache.org/docs/apache-airflow/2.3.4/start/docker.html

 

curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.3.4/docker-compose.yaml'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10221  100 10221    0     0  59012      0 --:--:-- --:--:-- --:--:-- 59080

mkdir airflow
cd dirflow

echo -e "AIRFLOW_UID=$(id -u)" > .env
mkdir -p ./dags ./logs ./plugins

docker-compose up airflow-init

 

 

 

docker-compose up

 

 

docker ps

 

로그인 

localhost:8080

airflow / airflow

 

 

위 예제를 보고 싶지 않다면 docker-compose.yaml 에서 Airflow__core__load_examples 를 false 로 변경.

AIRFLOW__CORE__LOAD_EXAMPLES: 'true'

docker compose down --volumes --remove-orphans


포트변경 (docker-compose.yaml 에서 8080 -> 8085로 변경.

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - 8085:8080
    healthcheck:
    test: ["CMD", "curl", "--fail", "http://localhost:8085/health"]