Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
4.3k views
in Technique[技术] by (71.8m points)

Airflow - Xcoms and parallel jobs - problem (xcoms overwriting themselves)

I have the pipeline below. On my 2nd level of tasks (first paralel tasks) I write a Xcom with some ID, then in the 3rd (2nd col. in paralel) and 4th column of tasks (3rd in paralel) I retrieve the xcoms I wrote in column two. The problem is that since it's writing 6 tasks at the same time, the XCOM that sticks in the end is the last one that was written.

Is there a way for me to make these Xcoms work in parallel? by now because of this problem I have to run all of theses tasks linearly, so the xcoms work correctly... Image can be seen here: https://i.stack.imgur.com/aXFWz.png


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)
等待大神解答

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...