Skip to content

[Metrics] Frontend-exporter fault tolerance quick fix #429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dcrankshaw merged 19 commits into from
Mar 13, 2018
Merged

[Metrics] Frontend-exporter fault tolerance quick fix #429

Changes from 17 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
e6c8263
Fix dict unpacking lint error
simon-mo Jan 17, 2018
797f319
Format Code
simon-mo Jan 17, 2018
299491b
Update yapf
simon-mo Jan 17, 2018
7b54e6e
Update yapf and reformat code
simon-mo Jan 17, 2018
7a96dcf
Check yapf version in shell script
simon-mo Jan 18, 2018
65c7f80
Change `echo` to `cat`
simon-mo Jan 18, 2018
f3d5eac
Add python version check; almost ready
simon-mo Jan 18, 2018
ab9cc98
Format code with python version 2.7.12
simon-mo Jan 18, 2018
1aac3eb
Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…
simon-mo Jan 21, 2018
1bae380
Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…
simon-mo Feb 20, 2018
4a9d68a
Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…
simon-mo Feb 21, 2018
70b5e16
Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…
simon-mo Mar 1, 2018
e04711b
Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…
simon-mo Mar 7, 2018
092dd12
Merge branch 'develop' of https://github.com/ucbrise/clipper into dev…
simon-mo Mar 8, 2018
d4c73f0
Frontend-exporter fault tolerance
simon-mo Mar 8, 2018
8974138
Address Comment
simon-mo Mar 12, 2018
6ae7408
Address Comment
simon-mo Mar 12, 2018
6fb2120
Add logging basic config
simon-mo Mar 12, 2018
61a5f8d
Merge branch 'develop' into exporter-fix
dcrankshaw Mar 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions monitoring/front_end_exporter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from prometheus_client import start_http_server
from prometheus_client.core import GaugeMetricFamily, REGISTRY
import argparse
import logging

parser = argparse.ArgumentParser(
description='Spin up a node exporter for query_frontend.')
Expand All @@ -16,14 +17,21 @@
help='The name of docker container in clipper_network')
args = parser.parse_args()

Copy link
Copy Markdown
Contributor

@dcrankshaw dcrankshaw Mar 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add these lines right here:

logging.basicConfig(
    format='%(asctime)s %(levelname)-8s [%(filename)s:%(lineno)d] %(message)s',
    datefmt='%y-%m-%d:%H:%M:%S',
    level=logging.INFO)

logger = logging.getLogger(__name__)
query_frontend_id = args.query_frontend_name

ADDRESS = 'http://{}/metrics'.format(query_frontend_id)

logger.info("Scraping {}".format(ADDRESS))


def load_metric():
res = requests.get(ADDRESS)
return res.json()
try:
res = requests.get(ADDRESS)
return res.json()
except Exception as e:
logger.warning("Scrape Failed! Error: {}\n".format(e))
return dict()


def multi_dict_unpacking(lst):
Expand All @@ -38,6 +46,10 @@ def multi_dict_unpacking(lst):


def parse_metric(metrics):
if len(metrics) == 0:
# Return empty dictionary if it's empty
return metrics

wo_type = list(itertools.chain.from_iterable(metrics.values()))
wo_type_flattened = list(itertools.chain([flatten(d) for d in wo_type]))
wo_type_joined = multi_dict_unpacking(wo_type_flattened)
Expand Down