spring-cloud-dataflow-server-yarn/index.html at 02bc9547b59ec2bf6088c2df8caa4ef409ab9cc1 · spring-attic/spring-cloud-dataflow-server-yarn · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
---
# The name of your project
title: Spring Cloud Data Flow for Apache YARN

badges:

  # Specify your project's twitter handle, if any. Delete if none.
  twitter: SpringCloudOSS

  # Customize your project's badges. Delete any entries that do not apply.
  custom:
    - name: Source (GitHub)
      url:  https://github.com/spring-cloud/spring-cloud-dataflow-server-yarn
      icon: github

    - name: Issues (Waffle)
      url:  https://waffle.io/spring-cloud/spring-cloud-dataflow-server-yarn
      icon: tracking

    - name: CI (Bamboo)
      url:  https://build.spring.io/browse/SCD-YARNBMASTER
      icon: ci

    - name: StackOverflow
      url:  https://stackoverflow.com/questions/tagged/spring-cloud-dataflow
      icon: stackoverflow

---
<!DOCTYPE HTML>
<html lang="en-US">

<!-- Specify the parent of this project (or delete if none) to influence the rendering of the breadcrumb -->
{% capture parent_link %}
[Spring Cloud]({{ site.projects_site_url }}/spring-cloud)
{% endcapture %}


{% capture billboard_description %}

Spring Cloud Data Flow for Apache YARN is a cloud-native orchestration service for composable
data microservices on [Apache YARN](https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html). With Spring Cloud Data Flow,
developers can create and orchestrate data pipelines for common use cases such as data ingest, real-time analytics, and data import/export.

{% endcapture %}

{% capture main_content %}

Spring Cloud Data Flow for Apache YARN offers a collection of patterns and best practices for data
microservices running as streaming and batch data pipelines in Apache YARN.

## Features

* Consume streaming and batch data microservices as maven artifacts
* Create, unit-test, troubleshoot and manage data microservices in isolation
* Develop using: DSL, Shell, REST-APIs, Dashboard, and Flo
* Take advantage of Apache YARN value-adds such as metrics, security, logging, health checks, and remote management at each data microservice level
* Scale stream and batch pipelines through Spring Cloud Data Flow's `yarn-cli`

<span id="quick-start"></span>

## Quick Start

Step 1 - [Download](https://docs.spring.io/spring-cloud-dataflow-server-yarn/docs/current-SNAPSHOT/reference/htmlsingle/#_download_and_extract_distribution) Spring Cloud Data Flow's Apache YARN Server and Shell Applications

Step 2 - Start the [necessary peripherals](https://docs.spring.io/spring-cloud-dataflow-server-yarn/docs/current-SNAPSHOT/reference/htmlsingle/#_prerequisites)

Step 3 - Start Spring Cloud Data Flow's [YARN Server](https://docs.spring.io/spring-cloud-dataflow-server-yarn/docs/current-SNAPSHOT/reference/htmlsingle/#_start_server) Application

Step 4 - [Run and Connect](https://docs.spring.io/spring-cloud-dataflow-server-yarn/docs/current-SNAPSHOT/reference/htmlsingle/#_connect_shell) the Shell Application

Step 5 - Register [Stream and Task](https://docs.spring.io/spring-cloud-dataflow-server-yarn/docs/current-SNAPSHOT/reference/htmlsingle/#_register_applications) applications

Step 6 - Create **‘ticktock’** Stream
```
dataflow:>stream create ticktock --definition "time | hdfs --rollover=1000" --deploy
```

Step 7 - Verify **‘ticktock’** Results under `/tmp/hdfs-sink/` folder

Step 8 - Launch Dashboard at: http://&lt;SCDF_SERVER_HOST&gt;:&lt;PORT&gt;/dashboard

{% endcapture %}

{% capture related_resources %}

### Related Projects

* [Spring Cloud Data Flow](https://cloud.spring.io/spring-cloud-dataflow/)
* [Spring Cloud Stream](https://cloud.spring.io/spring-cloud-stream/)
* [Spring Cloud Stream Applications](https://github.com/spring-cloud/spring-cloud-stream-app-starters)
* [Spring Cloud Task](https://github.com/spring-cloud/spring-cloud-task/)
* [Spring Cloud Task Applications](https://github.com/spring-cloud/spring-cloud-task-app-starters)
* [Spring Cloud Data Flow Samples](https://github.com/spring-cloud/spring-cloud-dataflow-samples)
* [Spring Cloud Data Flow UI](https://github.com/spring-cloud/spring-cloud-dataflow-ui)

{% endcapture %}


{% include project_page.html %}
</html>