Help your Enterprise Implement Big Data with Control-M for Hadoop
•Als PPTX, PDF herunterladen•
3 gefällt mir•2,567 views
Get better results from your Big Data. Learn how to build a better Hadoop business service using BMC ControlM for better service quality, and increased business agility.
Empfohlen
Empfohlen
Weitere ähnliche Inhalte
Was ist angesagt?
Was ist angesagt? (20)
Andere mochten auch
Andere mochten auch (20)
Ähnlich wie Help your Enterprise Implement Big Data with Control-M for Hadoop
Ähnlich wie Help your Enterprise Implement Big Data with Control-M for Hadoop (20)
Mehr von BMC Software
Mehr von BMC Software (17)
Kürzlich hochgeladen
Kürzlich hochgeladen (20)
Help your Enterprise Implement Big Data with Control-M for Hadoop
- 1. © Copyright 1/15/2015 BMC Software, Inc1 Joe Goldberg Solutions Marketing October 2014 Help your Enterprise Implement Big Data with Control-M for Hadoop
- 2. © Copyright 1/15/2015 BMC Software, Inc3 Who is using Hadoop and why?
- 3. © Copyright 1/15/2015 BMC Software, Inc4 Who is using Big Data? $ • New Account Risk Screens • Fraud Prevention • Trading Risk • Insurance Underwriting • Accelerate Loan Processing Financial Services • 360° View of the Customer • Analyze Brand Sentiment • Personalized Promotions • Website Optimization • Optimal Store Layout Retail • Call Detail Records (CDRs) • Infrastructure Investment • Next Product to Buy (NPTB) • Real-time Bandwidth Allocation • New Product Development Telecom Healthcare • Genomic data for medical trials • Monitor patient vitals • Reduce re-admittance rates • Store medical research data • Recruiting for pharmaceutical trials Utilities, Oil & Gas • Smart meter stream analysis • Optimize lease bidding • Compliance reporting • Proactive equipment repair • Seismic image processing Public Sector • Analyze public sentiment • Protect critical networks • Prevent fraud and waste • Crowdsourced reporting • Fulfill open records requests
- 4. © Copyright 1/15/2015 BMC Software, Inc5 Hadoop Lots of Data Traditional
- 5. © Copyright 1/15/2015 BMC Software, Inc6 The Players
- 6. © Copyright 1/15/2015 BMC Software, Inc7 Building a (Hadoop) Business Service
- 7. © Copyright 1/15/2015 BMC Software, Inc8 • Identify Data Sources and Targets • Write code • Test • Deploy • Production The Steps
- 8. © Copyright 1/15/2015 BMC Software, Inc9
- 9. © Copyright 1/15/2015 BMC Software, Inc10 SQL Data Source • SQL Server – Write SQL script/Stored Procedure – Learn SQL Agent Job Definition – Write powershell script/bat file – Define job – Run job
- 10. © Copyright 1/15/2015 BMC Software, Inc11 ETL Data Source • Informatica – Build Informatica workflows – Learn PowerCenter Scheduler – Write Scripts – Build PowerCenter job – Run job
- 11. © Copyright 1/15/2015 BMC Software, Inc12 Files Data Source • Move files with FTP – Learn FTP tool – Write scripts – Run FTP
- 12. © Copyright 1/15/2015 BMC Software, Inc13 Run Hadoop Jobs • Oozie/Hue – Write the MapReduce, Pig – Learn Oozie – Write scripts – Build workflows
- 13. © Copyright 1/15/2015 BMC Software, Inc14 SQL Query #!/usr/bin/sh # Sample pmcmdscript set pagesize 0 linesize 80 feedback off SELECT 'The database ' || instance_name || ' has been running since ' || to_char(startup_time, 'HH24:MI MM/DD/YYYY') FROM v$instance; SELECT 'There are ' || count(status) || ' data files with a status of ' || status FROM dba_data_files GROUP BY status ORDER BY status; SELECT 'The total storage used by the data files is ' || sum(bytes)/1024/1024 || ' MB' FROM dba_data_files; #!/usr/bin/env bash bin=`dirname"$0"` bin=`cd "$bin"; pwd` . "$bin"/../libexec/hadoop-config.sh #set the hadoopcommandand the path to the hadoop jar HADOOP_CMD="${HADOOP_PREFIX}/bin/hadoop --config $HADOOP_CONF_DIR“ #find the hadoop jar HADOOP_JAR='‘ #find under HADOOP_PREFIX (tar ball install) HADOOP_JAR=`find ${HADOOP_PREFIX} -name 'hadoop--*.jar' | head -n1` #if its not found look under /usr/share/hadoop(rpm/deb installs) if [ "$HADOOP_JAR"== '' ]then HADOOP_JAR=`find /usr/share/hadoop-name 'hadoop--*.jar' | head -n1` fi #if it is still empty then dont run the tests if [ "$HADOOP_JAR"== '' ]then echo "Did not find hadoop--*.jarunder '${HADOOP_PREFIX} or '/usr/share/hadoop'" exit 1 fi #dir where to store the data on hdfs. The data is relative of the users home dir on hdfs. PARENT_DIR="validate_deploy_`date+%s`“ TERA_GEN_OUTPUT_DIR="${PARENT_DIR}/tera_gen_data“ TERA_SORT_OUTPUT_DIR="${PARENT_DIR}/tera_sort_data“ Hadoop #!/bin/ksh cd /home/bmcU1ser/ftp_race_source sftp -b /dev/stdin -o Cipher=blowfish -o Compression=yes -o BatchMode=yes -o IdentityFile=/export/home/user/.ssh/id_rsa -o Port=22 bmcUs1ser@hou-hadoop- mstr 1>sftp.log 2>&1 <<ENDSFTP if [ -f /home/bmcU1ser/ftp_race_target/daily_shipment_log]; then exit 1 else put daily_shipment_log/home/bmcU1ser/ftp_race_target fi quit ENDSFTP rc=$? if [[ $rc != 0 ]]; then print "***Erroroccurred...$rc" `date "+%Y-%m-%d-%H.%M.%S"` if [[ -f /home/bmcU1ser/ftp_race_target/daily_shipment_log ]]; then rm /home/bmcU1ser/ftp_race_target/daily_shipment_log fi else mv /home/bmcU1ser/ftp_race_source/daily_shipment_log /home/bmcU1ser/ftp_race_source/old/daily_shipment_log print "***Successful transfer...$rc" `date "+%Y-%m-%d-%H.%M.%S"` fi File TransferInformatica #!/usr/bin/bash # Sample pmcmdscript # Check if the service is alive pmcmd pingservice -sv testService -d testDomain if [ "$?" != 0 ]; then # handle error echo "Could not ping service" exit fi # Get service properties pmcmd getserviceproperties -sv testService -d testDomain if [ "$?" != 0 ]; then # handle error echo "Could not get service properties" exit fi # Get task details for session task "s_testSessionTask" of workflow # "wf_test_workflow" in folder "testFolder" pmcmd gettaskdetails -sv testService -d testDomain -u Administrator -p adminPass -folder testFolder -workflow wf_test_workflow s_testSessionTask if [ "$?" != 0 ]; then # handle error echo "Could not get details for task s_testSessionTask" exit fi Programmers program
- 14. © Copyright 1/15/2015 BMC Software, Inc15 What happens when this runs? • What is related to what? • Are we on time or late? • What if something fails? – Which program was running? – Where is the output? – How do I fix it? – Can I just rerun it? If so, from the beginning? – Does any cleanup have to be done? – How do I track this problem and the steps taken to resolve the problem? #!/usr/bin/env bash bin=`dirname"$0"` bin=`cd "$bin"; pwd` . "$bin"/../libexec/hadoop-config.sh #set the hadoopcommandand the path to the hadoop jar HADOOP_CMD="${HADOOP_PREFIX}/bin/hadoop --config $HADOOP_CONF_DIR“ #find the hadoop jar HADOOP_JAR='‘ #find under HADOOP_PREFIX (tar ball install) HADOOP_JAR=`find ${HADOOP_PREFIX} -name 'hadoop--*.jar' | head -n1` #if its not found look under /usr/share/hadoop(rpm/deb installs) if [ "$HADOOP_JAR"== '' ]then HADOOP_JAR=`find /usr/share/hadoop-name 'hadoop--*.jar' | head -n1` fi #if it is still empty then dont run the tests if [ "$HADOOP_JAR"== '' ]then echo "Did not find hadoop--*.jarunder '${HADOOP_PREFIX} or '/usr/share/hadoop'" exit 1 fi #dir where to store the data on hdfs. The data is relative of the users home dir on hdfs. PARENT_DIR="validate_deploy_`date+%s`“ TERA_GEN_OUTPUT_DIR="${PARENT_DIR}/tera_gen_data“ TERA_SORT_OUTPUT_DIR="${PARENT_DIR}/tera_sort_data“ Hadoop
- 15. © Copyright 1/15/2015 BMC Software, Inc16 SQL Query HadoopFile TransferInformatica A Better Way
- 16. © Copyright 1/15/2015 BMC Software, Inc17 Defining Control-M for Hadoop jobs Set Script parameters Hadoop Program parameters HDFS commands • get • put • rm • move • rename Supports all Apache Distributions (0.x-2.x): • Cloudera • Hortonworks • MapR • Pivotal • BigInsights
- 17. © Copyright 1/15/2015 BMC Software, Inc18 Building a Hadoop Business Process HDFS Java MapReduce Pig Hive Sqoop File Transfer Informatica DataStage Business Objects Cognos Oracle Sybase SQL Server SSIS PostgreSQL z/OS Linux/Unix/Windows Amazon EC2 / VMware NetBackup / TSM SAP / OEBS / Peoplesoft
- 18. © Copyright 1/15/2015 BMC Software, Inc19 Connection Profile
- 19. © Copyright 1/15/2015 BMC Software, Inc20 Monitoring Workflows Resource Manager report
- 20. © Copyright 1/15/2015 BMC Software, Inc21 Workload Conversion & Discovery
- 21. © Copyright 1/15/2015 BMC Software, Inc22 BMC Control-M Workload Automation Hadoop Application Developers Write programs Build Hadoop jobs Add Pre/Post Jobs Access for the Business IT Scheduler Pig Hive MapReduce Sqoop HDFS File Watcher
- 22. © Copyright 1/15/2015 BMC Software, Inc23 And the fun is just beginning…
- 23. © Copyright 1/15/2015 BMC Software, Inc24 Partner Why BMC Control-M for Hadoop?
- 24. © Copyright 1/15/2015 BMC Software, Inc25 Key Takeaways Eliminate scripting with built-in capabilities Reduce the complexity of building and testing applications Production Applications run more reliably, are easier to monitor and ensure compliance Big Data and Hadoop are coming to your Data Center Easily build composite applications leveraging the full power of your technology fabric Build Applications Faster Increase Service Quality Gain Business Agility
- 25. © Copyright 1/15/2015 BMC Software, Inc26 For Additional Information: www.bmc.com/hadoop
- 26. © Copyright 1/15/2015 BMC Software, Inc27 Thank You. Be sure to visit Control-M Labs
Hinweis der Redaktion
- Every industry segment is already using Big Data. If yours is not in the above list, it’s only because there’s not enough room.
- This is a single-slide explanation and justification for Hadoop: Companies have lots of data both conventional structured data and new types. But the fundamental problem is there is more data than can be economically processed with traditional approaches. The traditional approach takes the ENTIRE data set and loads/copies into a relational database or some other file structure and processes it from top to bottom sequentially. This takes a long time and the hardware is VERY expensive. Even if money was no object, and when is THAT ever the case, getting bigger and bigger hardware would still give relatively incremental increases to capacity and speed. Hadoop on the other hand, uses a cluster of cheap (NOT inexpensive but cheap) hardware that is expected to fail and so is disposable. (The cluster technology provides full redundancy so that when (not if) a component fails, the switch to an alternate copy of the data being processed is completely transparent). The data is broken up into as many pieces as you have servers in the cluster (the largest Hadoop clusters have tens of thousands of nodes) and is processed in parallel. It is common for processing time to drop from hours or even days to minutes.
- So let’s talk about a project to deliver the first business service to take advantage of Big Data. You’ll see in a moment why (Hadoop) is in brackets.
- A business requirement/goal is established and Application Developers start designing. One of the first steps/requirements that we have heard from almost every single customer we’ve talked to is identifying the data that will be processed. Almost without exception, that data includes a whole bunch of traditional data sources like relational databases, data processed by ETL tools, data transferred from various sources and perhaps some new data types like social, web click or sensor data. Then all the other phases of Application Development are performed. This sounds very similar to other projects and in fact, much of what I’m about to discuss applies to just about every application development project.
- So this sounds easy, right? Just like the stuff you do all the time. Perhaps not. The pressure to deliver Big Data applications is huge. Many identify Big Data initiatives as key competitive differentiators and time is not your friend. Hadoop and Big Data are relatively new technologies so there are few experienced, seasoned practitioners. According to Gartner, in the next few years, the market will provide only 25% of the required staffing to fill Big Data positions. These factors conspire to make Big Data/Hadoop projects particularly challenging to staff and deliver and the common pitfalls and delays that are all too common among traditional projects make Big Data projects all the more difficult for organizations to pull off successfully. This “piece of cake” can make you very sick.
- Let’s examine some of the factors that are among the more challenging problems, especially because if you go down the common path, it becomes really difficult to change later. We’ve identified data sources. Let’s quickly look at how each one if handled within the context of our first Big Data project.
- So somebody in AppDev says “I know how to fix this” and scripts a bunch of this stuff, but not all of these tools can be eliminated so you get a bunch of scripts AND a bunch of tools. Testing is done and the application is delivered to Operations to “run this stuff”
- So somebody in AppDev says “I know how to fix this” and scripts all this stuff. Testing is done and the application is delivered to Operations to “run this stuff”
- There are five Control-M job types: Java MapReduce Pig script Hive Sqoop HDFS File Watcher All five are sub-types of the Hadoop job type. Select “Hadoop” from the Job Palette. Then select the “execution” type. CLICK You can add program and environment parameters for this specific execution CLICK It’s common to have to manipulate files before and after program execution so the Pre/Post Commands enable you to perform HDFS operations via the “Pre Commands” and Post Commands” sections in the job definition. You can also choose whether the success of these Pre/Post actions will affect overall job status or not.
- Once you have built your Hadoop jobs, building a flow for either just Hadoop jobs or connecting those Hadoop jobs into an enterprise business process that may include ETL, RDBMS extracts, File Transfers and any other job types/applications that Control-M supports, is a simple, drag and drop process. And when the workflow is finished, if you need to add an SLA or add a backup at the end or start up a VM that may not always be powered up, that too is a simple drag and drop addition of a BIM or Backup or Vmware job.
- The Connection Profile simplifies setup by collecting all environment info into a single object that is encrypted and managed by Control-M.
- Monitoring Hadoop is just like any other Control-M application with the ability to view job output, perform operational actions like Kill and provide visibility via Self Service.
- Control-M now provides huge value and great capabilities through the entire Lifecycle for Hadoop applications. Developers can build Control-M jobs with Workload Change Manager (the simple, web-based self-service job authoring component which you will hear about very shortly), submit requests to Production Control that enables that group to service the request quickly and get it into production and once in operation, provide access to all constituents in the business via Control-M Self Service.
- For Big Data, there seems to be a new project almost every day that will requires integration. The challenges I’ve been discussing are destined to be encountered over and over again for these and other technologies.